Communications

Broadband capacity - sizing buffers to handle traffic bursts

This is part 4 in a sequence of posts examining how broadband services actually work.  Part 1 looked at ISP concentration ratios.  Part 2 examined the effect of  averaging traffic from many subscribers.  Part 3 considered the impact of congestion on user experience. Here we'll take a finer grained look at traffic in the access network.

How do we achieve "zero" packet loss? Most websites are "hosted" and connect to the backbone by Fast Ethernet or Gigabit Ethernet links. Individual websites may be mismanaged, but there is no excuse for packet loss here.

Internet backbone links and backbone-to-backbone hand-offs are extremely high capacity multiplexing the traffic of thousands to millions of users.  Traffic statistics are predictable and operators compete on the basis of service levels, so packet loss is typically zero.

If there are performance problems, they're usually in the access network. Of course, one bottleneck is the service I've signed up for, for example a DSL link at 6 Mbps /1 Mbps. By definition my traffic is shaped to the service level I've signed up for.  This shaping typically happens at the first aggregation point (CMTS, DSLAM or WISP P2MP radio) or, on an IP or MAC address basis, at a bandwidth management device further upstream.  If they arise, problems are in the aggregation points and/or links between me and the backbone.

Since packet loss significantly degrades user experience, we'd like to understand how much over-provisioning a broadband service provider needs in order to absolutely minimize packet loss in the access network.  To understand this we need to look at smaller levels of aggregation (dozens and hundreds of users) and shorter intervals (sub-second rather than 5 minute averages, as few routers buffer more than a few tens or hundreds of milliseconds of traffic).

But first lets look at some five minute averages from a WISP with ~300 customers, each with either 1 Mbps or 3 Mbps service.

If one took these graphs at face value, it seems 20 Mbps of backhaul would be more than adequate, as the traffic peaks at 17 Mbps.  But 5 minute averages don't speak to what's really happening.

I don't have sub-second averages for the specific WISP above, but Fergal Toomey, Chief Scientist at Corvil Ltd., has some very detailed measurements on a similar Internet access network providing services to users at 512 Kbps, 1 Mbps and 2 Mbps. These are available in a whitepaper you can request from Corvil's website. Fergal's measurements were made at the Internet POP looking at traffic headed towards individual users.  Also, his measurements were made in 2004 when most hosted servers had 100 Mbps connections to the backbone and the majority of computers had TCP stacks configured for a maximum TCP receive window of 32KB.  In this graph, green shows 5 minutes averages, while light blue shows 500 ms averages. Dark blue shows 5 ms averages and they are striking!

What's Happening?

Web browsers typically use persistent TCP connections to minimize connection overhead and to avoid repeating TCP's slow start.  As a result, servers will frequently respond to a new request with a full window's worth of data.  In 2004 that was 32 KBs of data or ~23 packets of up to 1500 bytes each. Today it could be much more. Absorbing one such burst should be no problem, but to what extent are such bursts coincident?

Beyond web browsing, detailed traffic analysis uncovers a variety of odd behavior and some less savory behavior. For example, one burst was a blast of many 41-byte packets sent by a web application that apparently does a series of 1-byte writes to a TCP socket with the NO_DELAY flag set.  Several thousand packets arrive in quick succession causing packet loss.  Worse yet, follow-on losses occur as retransmissions cause a decaying series of echo bursts.

On a more malicious note, the data includes bursts caused by port scans.  These consist of many, many minimum size probe packets that attempt to scan all of a subscriber's ports in one enormous burst.  Luckily these lost packets are not retransmitted.

Beyond these, Fergal Toomey's analysis uncovered a variety of application anomalies that I found unexpected, but which crop up in the real world.

What headroom is really required?

Clearly we could just increase the size of the buffers in the router and smooth out any of these bursts, but that would also increase the delay while the burst lasts and thus introduce significant jitter.  A better question is how much headroom is required to guarantee zero packet loss and no more than 20 ms of incremental delay?

Conveniently, Fergal Toomey has run that calculation on the data set above and the result looks like this:

In this graph, the average traffic (the "mean" shown in light green) peaks just shy of 5 Mbps.  To guarantee zero packet loss and no more than 20 ms incremental delay 100 % of the time requires 19 Mbps of capacity or nearly 4x.  However, to guarantee zero packet loss and no more than 20 ms of delay 99% of the time, requires only 8 Mbps, i.e. a little less than 2x.

Interestingly, while ISP practices tend to be trade secrets, the rule of thumb that 2x headroom provides good service crops up repeatedly.

Conclusion

As a practical matter, real Internet traffic is fairly complex and includes a wide variety of anomalous behaviors. Rules of thumb based on averaging the traffic of a few hundred or even a few thousand subscribers are likely to be very expensive (in terms of excess capacity).  A better approach is to continuously monitor packet loss and delay variation (jitter), adding capacity as needed to keep those measures below desired levels.  

Zero packet loss and less than 20 ms incremental jitter, more than 99 percent of the time sounds plausible objective to me.  

Now if only my broadband service provider was willing to share any information about their service level objectives or performance.  Ooops, I must be dreaming.

Broadband capacity - the impact of congestion

This is part 3 in a sequence of posts examining how broadband services actually work.  Part 1 looked at ISP concentration ratios and part 2 examined the impact of averaging many subscribers. Future posts will consider how to size backhaul links and how to configure buffers.

Two major reasons people purchase broadband (or sign up for more capacity) are to improve interactivity and to get access to new applications, for example streaming movies.  New apps is clear, but it's worth looking congestion and how congestion affects interactivity.

Congestion has two impacts. First you get increased delays, as packets are buffered in queues waiting for free capacity on the next link. Usually this adds tens or hundreds of milliseconds to round trip times. Then, if the average traffic exceeds link capacity, you get packet loss. If you do have congestion, it's these packet losses that cause TCP traffic flows to back off, thus avoiding congestion collapse (and even less throughput).

Here is a graph of packet loss on a service that most of us would consider unacceptable.  These measurements were taken by Tom Dunigan on an early cable modem service (February 2001) that was substantially upgraded about a year later.

 

The next graph (also from Tom Donigan) shows how this congestion impacts interactivity.  The graph shows the echo delay for typing into a remote program using telnet in character-at-a-time mode.  One test (red line) was run in the early morning and one test (green line) was run during the evening when packet loss was peaking.  Each test measured the echo response time for 100 successive individual characters. Notice the typical echo delay goes up from ~130 ms in the morning to roughly 200-300 ms in the evening, but with occasional delays of more than one second.

Relatively few programs operate character-by-character, so the extra 100 ms or so might not matter.  On the other hand, a once second delay is noticeable!

A more common activity is viewing content on websites.  Here, the dominant time in any interaction is the time spent waiting for web content to download to your browser.  Since web content is downloaded using TCP (and more generally, TCP is the dominant protocol in use today), it's worth looking at the impact of packet loss on TCP throughput.  

The TCP protocol includes a congestion avoidance algorithm which is triggered by packet loss. When a packet is lost, the TCP sender slows down. As a result, the data rate for a single TCP flow looks like this (thanks to Guido Appenzeller):

Of course this assumes the TCP flow lasts long enough to saturate the bottleneck link and that this is the only flow in the network!  

What happens in real networks is complex, but RFC 3155 has an approximate formula for how a single flow is affected by packet loss in real network.  Even better, Bill Gibson at Niwot Networks has produced a nice graphic based on that formula for his paper on TCP limitations on file transfer performance.  It looks like this:

 

This is dramatic!  The different lines reflect different end-to-end round trip times (RTT) - times that will vary depending on the site you are connecting with.  Also, they represent the maximum throughput you could achieve with long duration TCP flows and no other bottlenecks.  What's notable is the logarithmic throughput scale on the left and the fact that, at 1% packet loss (0.01 on the bottom scale), potential throughput drops by a factor of 100!

Again, there are many caveats.  A real network has a mix of short- and long-lived flows. The local operating system may not be optimized to take full advantage of broadband speeds (although MS Windows actually got better at this with Vista). None-the-less, even 1% or 2% packet loss is correlated with poor user experience.

In short, to actually obtain the instantaneous throughput you thought you were purchasing, you don't want packet loss in upstream portions of the network.

I'll discuss what this actually implies for routers and backhaul links in a subsequent post.

Broadband capacity - the impact of averaging many subscribers

The fundamental idea of packet networks like the Internet is to transport variable rate data flows over a shared infrastructure.  Individual end points may generate little or no traffic for minutes or hours on end and then suddenly want to send or receive a large data set as quickly as possible.  Packet networks efficiently share common transport facilities among multiple users but how does this sharing actually work?  Yesterday I posted some actual, relatively current data from three small ISPs.  Today, let me show you how averaging traffic from many individual end points produces a completely stable core network flow.

Each of the graphs below shows successive five minutes measurements during a 24 hour period, but the first graph reflects a few hundred subscribers while the final graph represents the average of many, many millions of subscribers.

In the first graph, the blue line reflects outbound traffic from a few hundred subscribers.  While the average is just over 1 Mbps, the peak (at ~1615 hrs) is 6.9 Mbps for a peak-to-average of nearly 7-to-1.  The green bars represent inbound traffic with an average of 5.6 Mbps and a peak of nearly 16.8 Mbps for a peak-to-average of 3-to-1.  Some of this is time-of-day dependency but a lot of it, particularly the 6.91 Mbps spike at 1615 hrs, is the result of averaging only a few hundred subscribers.

 

Here's a traffic measurement at AMS-IX, the largest Internet Exchange in the world.  With an average of nearly 1 Gbps, the peak-to-average is a modest 1.43-to-1 and it's almost entirely due to time of day.  The largest change from one 5 minute measurement to the next appears to be about 140 Mbps so the short term peak-to-average is perhaps 1.1-to-1.

 

Finally, here a measurement of all the traffic flowing through AMS-IX – over 822 Gbps peak.  Here the time-of-day variation remains but there are virtually no statistical fluctuations.  The short term peak-to-average is almost 1-to-1.

 

To put this in economic terms, for an Internet backbone link that runs at many Gbps your daily traffic profile is completely stable and you can guarantee zero packet loss merely by providing 10%-20% extra capacity above your daily peak.  

But if you are running a small ISP, both the capacity you need per subscriber and the extra "headroom" for unanticipated peaks must be substantially larger.  To get a handle on what is required we also need to look at shorter intervals (shorter than five minutes).  But more on that in a subsequent post.

ISP Contention Ratios and average traffic per user – some data points

Recently there’s been a flurry of discussions about bandwidth shortages, not to mention bandwidth hogs, bandwidth caps, network neutrality and what constitutes fair practices by ISPs. But there is very little public data and no transparency! 

So let me offer a (very) few actual data points, one from a few years ago and three fairly current. These come from friends who are operators and from emails shared on trade association mail lists. Since some of it was not intended for publication, I have withheld specific names. 

Notice contention ratios vary widely and yet each of these operators has enough extra capacity that they claim to avoid link saturation and its impacts, like packet loss and excessive ping times. It’s true these are all small operators (280-1200 customers) so this data lacks the statistical smoothness one would expect when aggregating traffic from thousands or millions of subscribers. 

I also have some data on traffic averages by time and by number of subscribers that I’ll share in a subsequent blog post. 

Mar 2006 – Rural ISP (Dialup, DSL & Wireless)
130 Motorola Canopy subscribers (with 512 Kbps service)
150 DSL subscribers (110 with 512 Kbps service; 40 at 1 Mbps)
100 dialup lines serving 800 customers (average 40 Kbps)
Daily traffic was saturating 4.5 Mbps (3 T1s) for an hour per day but not with 6 Mbps.
Contention ratio: 164Mbps / 6 Mbps -> 27:1
Traffic per BB user: 6 Mbps / 280 subscribers -> 21 Kbps 

Sep 2009 – Rural ISP (DSL & Wireless)
1200 customers, DSL subs at 1.5, 3 or 6 Mbps; wireless subs at 384 Kbps, 768 Kbps, 1 Mbps or 3 Mbps. I don’t have the mix, so I'm guessing the contention ratio.
Daily traffic peaks at around 35-40 Mbps; up to 150 Mbps Internet transit available.
Contention ratio: 2400 Mbps(?) / 150 Mbps -> 16:1
Traffic per BB user: 40 Mbps / 1200 subscribers -> 33 Kbps 

Sep 2009 – Community-run WISP
340 customers, mostly on 3-4 Mbps service
Daily traffic peaks around 17 Mbps; 45 Mbps of Internet transit available.
Contention ratio: 1190 Mbps / 45 Mbps -> 26:1
Traffic per BB user: 17 Mbps / 340 members -> 50 Kbps 

Sep 2009 – Rural ISP
370 customers: 300 Customers on wireless (1-3 Mbps) and 70 customers with FTTH (10/100 & up, but most have 10 Mbps peak). Monthly fees based on a 10 or 20 GB cap with extra charges for heavier use.
Daily traffic peaks at 13-14 Mbps; 20 Mbps of Internet transit available.
Contention ratio: 1200 Mbps (?) / 20 Mbps -> 60:1
Traffic per BB user: 14 Mbps / 370 subscribers -> 37 Kbps

Has AT&T Wireless data congestion been self-inflicted?

We've all heard or read stories about how iPhone usage has overloaded the AT&T Wireless network but it's likely at least some of their problems are the result of configuration errors ― specifically, congestion collapse induced by misconfigured buffers in their mobile core network.

In early September, David Reed sent this interesting message to the IRTF's "end-to-end" email list.  List members include some world experts on Internet protocols.  During the next couple of days, there were over 40 messages in related threads.  While some of these experts were over-thinking the problem, if you are patient enough to read through the many messages, what emerges is clear.  At least in the case David measured (from a hotel room in Chicago, while he had 5 bars of signal strength, using an AT&T Mercury 3G data modem in his laptop), the terrible throughput and extreme delays he experienced appear to result from overly large buffers in the routers &/or switches in AT&T's core network.  Note:  if you don't want to read all the list messages the short summary is:  >8 second pings times!  What's more the effect was bymodal:  either ping times under 200 ms, or over 5 seconds.

Recently I was talking with a friend whose company continuously operates (and monitors) multiple 3G data links on the Verizon Wireless, AT&T Wireless and Sprint  PCS networks.  His data shows periods when the round trip time for http requests goes over 8 seconds, on the AT&T Wireless network only!  I don't have a copy of his data that I can examine in detail, but when combined with David Reeds report, it certainly appears AT&T Wireless has configuration problems.  If you read on you'll see this may not be the result of gross stupidity, but someone at AT&T Wireless should be a little embarrassed.

My (techie outsider's) Analysis of What's Happening

Buffers in Packet Networks

Routers (& switches) in a packet network have to include buffering in order to absorb transient traffic bursts.  Unfortunately, despite decades of research and operational experience, there is no simple formula for how much buffering is optimal at any given location in a network.  If you're interested in more detail, Ravi Prasad has a good review of the literature on pages 10 & 11 of his (April 2008) PhD thesis.  But decades of operational experience have yielded some basic precepts and it appears AT&T Wireless is violating at least one basic precept.

The buffer in front of a congested link must induce some packet loss.  TCP (the dominant Internet protocol) continuously increases it's transmit rate until it experiences packet loss, then it cuts its rate in half and enters a congestion avoidance mode.  If the network becomes full but there is no packet loss, each TCP sender will keep increasing it's rate causing the network to suffer a congestion collapse.

In the case of a mobile network, the limited resource is over-the-air capacity.  Backhaul may also be expensive, but it's relatively easy to over provision anything else in the operator's network.  So the issue is, how big should the buffer be in the last router between the high capacity core network and the actual over-the-air data path to a subscriber?  Ideally we'd like enough buffer to absorb momentary packet bursts that, averaged out, don't exceed the available over-the-air capacity.  But as soon as the offered traffic exceeds the available over-the-air capacity, we want some packet loss.  The complicating factor is the way 3G wireless networks schedule over-the-air traffic.

Jitter in 3G radio networks

One cellular base station serves multiple users and the quality of the connection to any specific user depends upon instantaneous wireless propagation characteristics.  These can vary second by second and even millisecond by millisecond when a user is moving.  To deal with over-the-air losses, the basestation (the "Node-B" in a 3G network) keeps copies of each packet until a positive acknowledgement (ACK) is received, retransmitting the packet if the ACK is not received in time. Of course retransmissions introduce delay and jitter.  Furthermore, at any given instant, some users wireless links are better than others.  In order to maximize the total traffic in a cell, the 3G MAC layer schedules transmissions to individual users based on who has the best instantaneous throughput.  This is an efficient solution but it also introduces different amounts of jitter into each user's data path.  Luckily these effects are well understood and not that severe.  With HSDPA, the basic transmission time interval is 2ms so total delay variation is relatively small.  This graph from Jang et al (1) is typical of measured values in an HSDPA network.

Most jitter is below 15 ms.  Measurements of ping latency between 3G wireless devices and the first IP server at the edge of the mobile core network (typically the GGSN) can extend out to over a second as this graph from Mun Choon Chan and R. Ramjee (2) shows:

but most of the time, total IP latency is a few hundred milliseconds. As mentioned earlier, David Reed reported bimodal operation on the AT&T Wireless network with normal behavior yielding ping times under 200 ms.

Likely cause of AT&T's problems

So what is happening in the AT&T Wireless network when ping times go over 8 seconds?  We know how a customer's IP packets are passed through an operator's mobile core network.  They are tunneled all the way from the handset to the Gateway GPRS Signaling Node (GGSN), i.e. to the router where the mobile core network connects to other networks.  The protocol stacks for this tunneling look like this:

More recent versions replace the ATM and AAL5 with Ethernet and IP, but the user never sees this as user IP data is tunneled across the top of the diagram (carried by PDCP and GTP-U).  As a result, user traceroutes can't reveal the detail of what's happening in the core network.  The first thing the user can see is the GGSN (the gateway to the next network).  So we can't make conclusive measurements from outside the network, but we do know a few more things.

The bottleneck link is the over-the-air link, i.e. the connection from radio access network or UTRAN to the Mobile Statation (MS) in the above diagram, therefore the critical buffers are those at the UTRAN.  In practice the UTRAN includes both the basestations (called Node-Bs) and the Radio Network Controllers (RNCs) which coordinate handovers between basestations (among other things).  Because of hand-overs, the amount of data buffered at the Node-B is relatively small. It's the buffers at the RNC that must be large enough to deal with the delay variations in the radio network and yet small enough to induce packet loss when the network gets congested.

While I don't personally have experience managing a 3G HSDPA network, my impression is UTRAN buffers are normally less than 200 ms.  Recently Yerima and Al-Begain presented an interesting paper (3) on buffer management in 3.5G wireless networks in which they concluded that 120 ms buffers were ideal for downlink traffic in a specific UMTS-HSDPA configuration.

Zero Packet Loss

It appears AT&T Wireless has configured their RNC buffers so there is no packet loss, i.e. with buffers capable of holding more than ten seconds of data.  Zero packet loss may sound impressive to a telephone guy, but it causes TCP congestion collapse and thus doesn't work for the mobile Internet!

=================================================

References

(1) 3G and 3.5G Wireless Network Performance Measured from Moving Cars and High-Speed Trains, by Keon Jang† (keonjang@an.kaist.ac.kr), Mongnam Han† (hgma11@gmail.com), Soohyun Cho∗ (shcho1@kt.com), Hyung-Keun Ryu∗ (hkryu@kt.com), Jaehwa Lee∗ (jhlee@koren21.net), Youngseok Lee‡ (lee@cnu.ac.kr), Sue Moon† (sbmoon@kaist.edu)

  † Computer Science Department, KAIST

  ‡ School of Computer Science & Engineering, Chungnam National University

  ∗ KT Central R&D Laboratory
http://an.kaist.ac.kr/~sbmoon/paper/intl-conf/2009-micnet-3g.pdf

(2) TCP/IP performance over 3G wireless links with rate and delay variation,

by Mun Choon Chan and R. Ramjee, Bell Labs  (presented by Jinqiang Yang)
http://wwwcsif.cs.ucdavis.edu/~ghosal/Teaching/ECS289I/Lectures/jinqiang-yang-wirelessTCP.pdf

(3) Dynamic Buffer Management for Multimedia Services in 3.5G Wireless Networks,

by Suleiman Y. Yerima and Khalid Al-Begain

Proceedings of the World Congress on Engineering 2009 Vol I

WCE 2009, July 1 - 3, 2009, London, U.K.
http://www.iaeng.org/publication/WCE2009/WCE2009_pp857-862.pdf

Structural Bypass - my talk from eComm last spring

At Spring eComm 2009 in California in March, I gave a presentation entitled:  Structural Bypass - A Simple, Proven Path to "Real Broadband" that dealt with policy approaches for first mile fiber deployments.  The video has just been posted here.  Unfortunately, this is my face and the audio with my words.  The slides are not shown in the video, instead you have to look at the slides here.

My arguments focused on urban areas only, as the economics are different between urban, suburban and rural deployments.  You can't just lump together all areas in a country and draw reasonable conclusions.  Also, the only data I have is for urban areas.  And, as a simplification for this talk (I only had 15 minutes including Q&A), my metric of success was the consumer price for 100 Mbps symmetric Internet connectivity.

My points were:

• Structural separation produces better results than other approaches.
• Separation between dark fiber and everything else is more effective than separations at higher layers.
• There is a political path, which I call Structural Bypass, which has been used at least once (in Stockholm), to effectively achieve structural separation without explicitly fighting that battle.

If you are interested, I suggest you open the video in one browser tab merely for the purpose of listening to the audio.  Then view the slides in another browser tab.  The audio is of some interest, but my facial expressions are of questionable value.

eComm agenda is on-line ―Check out the speakers!

There's a fascinating agenda building for eComm in Amsterdam in October.  Here's the latest list of talks, but what's best is the set of speakers who are giving these talks.  I personally know quite a few and know of many more (who I look forward to meeting).  The group includes a preponderance of innovators -- new views on what's happening, new ideas for how to drive change.  If you don't read through this list of talks, look at the speakers list here.  Then come to Amsterdam and meet these people.

21st Century Economics: Lessons for Telcos - Umair Haque (Havas Media Lab)

Advances In Spectrum Transparency, Software Defined Radio/Cognitive Radio - Darrin M Mylet (Spectru-Station)

Almost all Marketing & Product Management of Telco Services is Wrong - Rudolf van der Berg (Logica)

Current Trends in Community Wireless Networks and Beyond - Aaron Kaplan (FunkFeuer)

Death of the Handset: Evolving from Mobile Devices to the Mobile Digital Life - Mark Rolston (frog Design)

Edge As Value - Value As Edge - Graham Brierton (Voicesage)

Enslaving Humans using Communications Technology for Fun and Profit - Thomas McCarthy-Howe (Jaduka)

Entrepreneurial Advantages with New Open-Source Technologies - Jay Phillips (Voxeo)

European Telecoms 2015: Silent Death or Generative Bazaar? - Julien Salanave (IDATE Telecoms)

Finding Disruption - Michael Jackson (Mangrove Capital)

Goodbye Minutes, Hello Moments - Martin Geddes (BT)

Google Wave - David Wang (Google)

How the "Internet of Things" will Change the Way we Connect - Alexandra Deschamps-Sonsino (Tinker.it)

How to get More Value out of Customer Interactions by Blending Online with Voice - Stefan Hopmann (Swisscom)

Humans as a Service: Abstracting Communications to Reach New Applications - Todd Landry (NEC Sphere)

Is LTE being Held Hostage by Ordinary Voice Telephony? - Dean Bubley (Disruptive Analysis)

Lifestyle Segmentation from Carrier Location and Call Data - Greg Skibiski (Sense Networks)

Lifestyle Segmentation from Carrier Location and Call Data - Tony Jebara (Sense Networks)

LTE - Long Term Employment or Less Than Expected? - Moray Rumney (Agilent)

Open Access Makes Economic Sense - Benoît Felten (Yankee Group)

Open Screen Project: Next Generation Contextual Applications - Andrew Shorten (Adobe)

Opening of the Terahertz Region - Robert Horvitz (Open Spectrum Foundation)

Opportunities in Post-Telecom Connectivity - Bob Frankston (Frankston Innovating)

Post Financial Trauma - How is the Telecom Value Chain Now Positioned? - James Enck (mCapital)

Redefining Gifts in the Digital Space - Katie Lips (Little World Gifts)

Slowing Down Communication: Designs Inspired by Quality, Intimacy, and Humanity - Stefan Agamanolis (Distance Lab)

Spectrum 2.0 – What's Really Happening? - Brough Turner (Ashtonbrooke)

Stealth Approaches to Legislating Open Spectrum - Brough Turner (Ashtonbrooke)

Technology and Biological Evolution: What This Means for Media and Communications Technologies - Tomas Rawlings (University of the West of England)

Telemedia Futures - Gerd Leonhard (MediaFuturist.com)

The Emerging Telecology of Social Networks and the Status Update - Stuart Henshall (Phweet)

The Future of Interconnection - Rudolf van der Berg (Logica)

The Global Battle for Communications Justice: An Open Spectrum Manifesto - Sascha Meinrath (New America Foundation)

The Next Wave of Communications Applications - Cullen Jennings (Cisco)

The Practical Edge of Speech Technology - Moshe Yudkowsky (Disaggregate)

Ubiquitous Voice over Broadband - is There a Future Role for the Smart Pipe? - Martin Taylor (MetaSwitch)

Unlicensed Spectrum: Future Regulation - Prof. William Webb (Ofcom)

Video Killed the Telephone Czar? - Jan Linden (Global IP Solutions)

When Will HD Voice Become a Reality? - Martyn Davies (Dialogic)

What Would Telephony be Like if we Designed it Today? - Matt Ranney (RebelVox)

If you've read this far, there's a 10% discount via this registration link - register here.  And for more information, visit http://eComm.ec.  This one conference is at the top of my list for:  interesting, productive and an excellent set of contacts.

Pouzin Society - organizational meeting, today

The most interesting data networking ideas I've come across in the past N years are in the recent book by John Day, Patterns in Network Architecture: A Return to Fundamentals.  John Day's fundamental insight is that data networking is a distributed version of inter-process communications (IPC).  Those of us who come at this from the computer and computer operating system side of the world are quite familiar with IPC within one computer running one operating system.

The Pouzin Society (named after datagram inventor Louis Pouzin) is a group interested in investigating the implications of John Day's architecture and how it might be used in developing viable solutions to the current Internet architecture crisis.

Here's a top level picture (from a talk by John Day).  If you don't buy the book, there is an interesting summary in a presentation here.

Viewing data networking as a distributed IPC function (DIF) does a number of things.

• It clears up names versus addresses in a fashion that makes sense to me.  The application asks for connection to another named application while node addressing remains internal to the DIF.  Indeed, the point of attachment address for one DIF is a "name" presented to a lower level DIF.
• It makes management explicit.  To even establish an IPC, we must ask the DIF for the other party by name and then get their permission to exchange information (i.e. to establish the IPC session).
• It simplifies layers.  There is only one kind of layer, the DIF, that is used recursively.  Successive levels in the recursion differ in scope, and likely in policies, but architecturally they are the same.

I'm still getting up to speed, but this appears to be a fundamental theoretical breakthrough.  There are a mix of academics and practical networkers in the room for the Pouzin Society kickoff, so hopefully some practical implications for the evolution of the Internet will emerge in time.

Google's Peering and Caching Strategy

This is a followup on yesterday's post about how little Google/YouTube pays for bandwidth. Google wants to promote peering with ISPs, so they give presentations at ISP meetings.  After reading yesterday's post, Alex Benik of Battery Ventures sent me a link to this presentation given by Google at a 2008 meeting of the Latin America and Carribean Internet Addresses Registry (LACNIC).

As expected, Google peers with as many relevant ISPs as possible.  For the ISP, peering with Google eliminates their upstream costs for traffic to Google.  Since Google represents a substantial volume of traffic for most ISPs, this is a big saving.   As of May 2008, Google was present in 33 public Internet exchanges around the globe, so major ISPs already have connections in places where they can peer with Google.  The minimum qualifications are 5 Mbps of Google traffic and the ability to interconnect using Gigabit Ethernet at one of these 33 major Internet exchange points.

Google Global Cache

What's interesting is Google's caching strategy.  Just as Akamai puts servers in ISP's local facilities, Google is providing a distributed cache for their content.  This available to larger ISPs and allows them to serve Google content directly at the edge of their networks, thus reducing traffic on the ISP's backbone network.  Here's a representative rack that Google provides to the ISP.

YouTube's fine - Analysts don't understand Internet peering

As widely reported, Credit Suisse analysts have estimated Google's YouTube may lose $470M in 2009 and more in the future.  However, their estimates say Google will pay $360M for bandwidth in 2009.  I don't know how Google figures their cost of bandwidth, but anyone who understands anything about Internet transit/peering knows Credit is way off base.

Google does not pay for Internet Transit the way most tier 2/3 ISPs or most content providers must.  The economics are simple.  If you are a Tier 2 ISP, you have to purchase Internet Transit services from a Tier 1 network to handle that customer traffic which goes off your network and for which you cannot make other arrangements.  The most notable 'other arrangement' is peering.  If you have significant traffic to/from another specific network, you and the other network can both save Internet Transit costs by exchanging traffic locally, i.e. peering.  Of course an enormous amount of your traffic is directed to Google.  If you have a presence in any data center where Google has a presence, you would love to peer with Google, as that saves an enormous amount on your payments for upstream Internet Transit.

A similar effect plays out among Tier 1 providers.  If one tier 1 network cuts a special deal with Google, Google routes all their traffic through this provider and suddenly the other tier 1 networks have large asymmetries in their tier 1 peering arrangements.  Either they also cut deals with Google or they have to renegotiate their tier 1 peering arrangements to pay for the traffic asymmetry (something that's highly unlikely!).  Google is the one with leverage here!

I don't know what, if anything, Google pays for bandwidth, but it's not paying $360M for Internet transit.  Sorry Credit Suisse, you better go back to analyzing derivatives, credit swaps and other purely financial plays.

Google does have costs.  They have data centers in many parts of the world and they have a private fiber backbone that interconnects their sites and connects their private network to many, many potential peering points.  Operating their private backbone is a real cost to them and I haven't examined their financial reports to see if there is any way (from public data) of estimating their costs for this private network.

But until someone does this analysis, forget what you've read from Credit Suisse.

40th anniversary of RFC 1

April 7, 1969 was the day a graduate student at UCLA, Steve Crocker, published RFC 1 to the somewhat informal Network Working Group associated with the ARPA Network project.  I have heard Steve Crocker speaking informally about those days and my understanding is he created what became the "Request for Comments" series because anything more formal would have brought down several layers of governement bureaucracy.

It's interesting that the Internet owes its origins to a government project, but a lot of what happened was done by individuals and groups working around and/or in spite of government processes.

More bandwidth ― less delay, less latency

I was a little sloppy yesterday and several people have questioned my comment about latency.  I was reacting to slide by Herman Wagter of Amsterdam's Citynet in his presentation at F2C 2009.  His slide said:

Latency is the cause, bandwidth is the effect.

From his discussion it was clear he meant "Latency is the cause, bandwidth is the cure."  At the time he was talking about real-time person-to-person communications and illustrating it with a housebound person in Amsterdam who wanted to play cards with friends in other places (not nearby).  Verbally he mentioned the issue of sending large files.  In short he was addressing the real reason for high capacity Internet access links.

Why people want more "bandwidth"

It's not because they need or want to send and receive 100 Mbps of data all the time or even a significant part of the time.  The issue is delay, specifically serialization delay.  If I have a 1 Mbps upstream Internet connection and I want to send an email with a 5 MB Powerpoint file attached, it will take more than 40 seconds (5 MB ~= 40 Mbits).  On a 100 Mbps link, the same email is delayed only a fraction of a second.

Serialization delay also effects media streams, although much less.  If I want to send 500 Kbps of continuous video over that 1 Mbps uplink, serialization delay will cause added latency.  IP is a packet protocol and the 500 Kbps video stream will be broken into a stream of packets, typically ~1500 bytes (12 Kbits) each. While the serialization delay is only 12 ms on the first link, there is serialization delay on every link.  If there is another 1 mbps link at the other end, that's another 12 ms of delay.  And here, 12 ms is significant.  For a natural interaction between two people, you'd like to keep the round trip delay below 200 ms.  Nothing goes faster than the speed of light so transoceanic links introduce many 10s of ms of delay, each way.  It is very easy to eat up a 200 ms budget, so saving 12 ms at each end is significant.

Burst rate versus continuous

For me and for most people, the issue that drives demand for high speed access links is delay, not the amount of information that is to be sent or received.  Indeed, I'd love a service offered only a few Mbps average over a month, if I could be guaranteed 1 Gbps on a burst rate basis whenever I wanted.

David Weinberger is blogging F2C rather completely

And David is a expert blogger!  i.e., really good coverage.
  http://www.hyperorg.com/blogger/

The first Freedom to Connect conference entry is here. 

Latency is the cause; bandwidth is the solution

Actually, what Herman Wagter of Amsterdam's Citynet said at F2C 2009 was:  latency is the cause, bandwidth is the effect.  But his explanation matched my title above.

If you are attempting to interact with other people, whether by VoIP or just playing cards together (with video) you need less than 200 milliseconds of end-to-end delay.  If it's playing cards together, with video, and you need to exchange 500 Kbps in less than 200 ms, you need a 100 Mbps pipe!

It's latency that drives the need for high bandwidth.  Most people won't fill that pipe most of the time, but they need the pipe to guarentee that what they do send gets through rapidly.

Highlights from Day 1 of F2C

Shamelessly ranked by my areas of interest...

Tim Nulty of East Central Vermont Community Fiber.  Tim is a veteran network builder and a forceful speaker, so he's happy to tell it like is.  He's also got that Yankee mix of liberal politics with extreme fiscal conservatism.  He's building fiber networks, in rural Vermont, which pay for themselves.

Ken Biba of Novarum has been measuring actual wireless networks in buildings and in cities for years.  While the detail is in a report available for purchase, the summary is that WiFi-based Muni WiFi yields significantly better performance than 3G cellular.  Interestingly coverage and reliability is right up there in selected cities, as well.  The take-away - 802.11n really rocks. I.e., the next cycle of WiFi is going to be vastly better than what he's been measuring over the past 3 years.

Ellen Miller of Sunlight Foundation was low key by comparison with Tim or Ken, but her stories were compelling - multiple instances of Internet community feedback creating the kind of information that the "open government" initiatives aspire to.

Finally, Dewayne Hendricks is always interesting.  This year he seemed more optimistic than last, presumably the result of the recent election.  In any event, here's another speaker with deep experience in building networks.

Freedom to Connect 2009 and Municipal Fiber

I'm attending F2C 2009 in Silver Spring Maryland. If you are here, please say hello.

Things are just getting started with a panel on Municipal Networks, led by Joanne Hovis, President of Columbia Telecommunications Corporation (CTC).  Panelists are:

Tim Nulty, formerly of Burlington Telecom and now running East Central Vermont Community Fiber, has tons of experience building fiber networks in low density areas (Vermont).

Dirk van der Woude, from Amsterdam's CitiNet, to talk about the Amsterdam's municipal fiber to the home project..

Lev Gonick, CIO Case Western Reserve, who was a key player in creating a 4000 mile fiber network for Cleveland and northeast Ohio under a community organization called OneCommunity.

Bill Schrier, CTO for Seattle, which is starting a fiber project, but already has it's own electric power utility.  (Although Bill implies they have had to drag their utility brethren into this).

What's interesting is the discussion on the chat backchannel is not about muni vs. commercial, but wireless versus fiber.

Tim Nulty has a strong argument that wireless is excellent for mobility, but not economical for fixed access.  In rural Vermont, a WiMAX network would cost $35M if you could get access to the spectrum (which is being horded by others).  Fiber would cost $70M but has 50 times the capacity and several times the revenue potential versus the wireless approach.  Further, if you deploy wireless as an addon after you have the fiber network (and the customer support infrastructure), the incremental cost is dramitcally less (perhaps $10M) and you get enough incremental revenue to get a good return on investment.  IN other words, you make more money if you do fiber first.  Tim's key to success is to get as near complete deployment as possible - something that is possible in areas where the incumbents are going.  Second, he goes for community ventures as a way to qualify for muncipal bonds.

Shareholders should demand phone companies open their fiber networks

A few weeks ago, Benoît Felten of Yankee Group and Fiberevolution gave a speech at a New Zealand Commerce Commission event which included a fascinating argument.

He points out that receiving a return on the substantial capital investments that a fiber to the home (FTTH) project requires is much more dependent on takeup rates than on the average revenue per user.  

Single player large scale deployments usually achive only 20%-25% initial adoption after which growth is exceedly slow.  On the other hand, systems which are open to competitors, i.e. with viable wholesale services, attract many more players who market, sell and deliver new services thus dramatically increasing adoption and accelerating the wholesaler's return on investment.

In short, obtaining a regulatory holiday so you can run your new fiber network as a monopoly is actually bad for your shareholders!

Benoît's speech was filmed.  For the full presentation (in four segments of < 10 minutes each on YouTube) see Benoît's post.

If you want just the essence, listen to the first three minutes of this:

Here is an enlarged version of the chart that Benoît is using while he speaks:

Gold bits and brown bits ― When you need QoS, two classes will do it!

Quality of Service is a contentious subject.  Every now and again, I look at Wikipedia's QoS page.  (Here are my comments 18 months ago).  While the Wikipedia content continues to evolve, it always downplays anything that questions the value of sophisticated QoS.  As of today, the section titled "QoS Problems" has been cut back to a single sentence (which does reference a classic anti-QoS paper):  "Internet2's QoS Working Group concluded that increasing bandwidth is probably more practical than implementing QoS."

Rather than directly debunk QoS yet again, let me investigate where QoS is actually useful by looking real life situations and the deployed solutions that allow for robust services despite congestion.  Identifying common elements in successful deployed solutions is much more realistic than theorizing about NGNs that still await deployment.

The first thing to notice: the only problem is on access links.  As I've written before:

Once you get beyond the access network, every link in the Internet — local, regional, national or international — is carrying multiplexed traffic from many users. Multiplexing many, many, bursty flows results in relatively predictable volume. Traffic volumes vary by time of day, but these links don’t saturate, except as a result of poor engineering or forecasting on the part of an ISP, or failures in other parts of the network causing rerouted traffic. Either case generates a rapid response from any ISP that expects to remain in business. So “best efforts” in the Internet core means sub-millisecond delay variations and near zero lost packets.

Net, net:  you never need QoS priority schemes in the core, all traffic gets excellent service. 

Access links are another story for two reasons.  First it can be very expensive to add more bandwidth.  This may be a byproduct of political/ regulatory issues, but we won't go there, at least in this post. :-)  Second, an access link may saturate due to an outgoing email with large Powerpoint attachment (the home case) or an equivalent event in the corporate environment, for example, transfer of a large CAD file.

But these are solved problems with multiple commercial products on the market for residential and corporate use.  It's worth looking at what these products do in order to understand exactly what kind of QoS is really required, when you can't just provision more bandwidth.

In the corporate market, there are two approaches.  The brute force approach splits voice and video telephony traffic from data traffic using two separate Internet connections with the VoIP connection sized to preclude any potential congestions.  When a single access link is shared, the typical solution employs a QoS router at the customer’s end. The QoS router gives absolute priority to outbound VoIP packets and protects inbound VoIP packets by active traffic shaping; i.e., by signaling remote TCP hosts to throttle inbound TCP data flows so there’s enough capacity for inbound VoIP packets.

Elsewhere within corporate LANs, there is either no QoS or simple priorities. Since VoIP traffic is a tiny percentage of all traffic, it can be given absolute priority without noticeably impacting other applications.  Thus some enterprise VoIP installations use QoS based on DiffServ or Ethernet 802.11p/q.  Typically there are just two classes assigned, with absolute priority for VoIP and video telephony.

I've written about equivalent solutions for residential situations that support VoIP and interactive gaming.  The solution?  Consumer VoIP phone adapters incorporate simple priority and have the ability to fragment large packets (so as to reduce serialization delays).  Because it's so useful for VoIP and also for interactive gaming, this functionality is showing up in popular residential routers from Linksys, Netgear, etc.

Two classes and simple priority is what it takes ― Gold bits for voice (and video and interactive gaming); Brown bits for everything else.

And that's only when you can't get more bandwidth.  On your own premises, it's likely cheaper and easier to upgrade to Fast Ethernet or Gigabit Ethernet.

Customer owned fiber in Ottawa

Arstechnica had a nice article yesterday by Timothy Lee entitled The Long Tail following up on Derek Slater's article last week on the Google Public Policy Blog entitled What if you could own your Internet connection?  Both articles are about a pilot project in Ottawa.

The "tail" in Timothy's article is the "last mile" (or as I prefer, "first mile") fiber connection from individual homes to a network peering point or other aggregation point where individuals can then choose from among multiple competing ISPs.  The importance is, as Timothy Lee puts it,

Without the expense of rolling out last mile infrastructure to every home, many more ISPs could afford to serve a given neighborhood by running wiring to the peering point, leading to more competition and lower prices. Perhaps best of all, the growth of customer-owned fiber could make debates over "open access" and network neutrality moot, as robust telecom competition should prevent the worst of the monopolistic behavior exhibited by telco and cable incumbents.

Typical costs run below $3000 per household depending on density and the number of households that agree to participate.  Considering the cost of a home, this is one investment that should easily increase the value of the home, most likely by much more than the cost!

Mentioned by Derek although unmentioned by Timothy is the most important person here, Bill St. Arnaud, Chief Research Officer at CANARIE, a Canadian nonprofit that focuses on network infrastructure.

Bill is driving the Ottawa trial and is the leading proponent of condominium fiber as described in his presentations here.  His work has also had a major impact on my thinking about "layer zero" competition.

Either way, articles on the Google blog and at Arstechnica will help promote this excellent concept and, hopefully, improve the quality of discussion in policy circles.

3G's biggest success is as a dumb pipe

Thanks to Dean Bubley for pointing out this presentation by Antero Kivi on the use of 3G mobile networks in Finland.

92% of all data traffic is from PCs

Furthermore, of the 4% of traffic from Symbian devices, 79% of that is for web access and 10% is for email.  So users are paying for mobile data for the sole purpose of connecting to the Internet.

Put another way, mobile operators have failed to provide any operator-specific data application of any relevance.

Why invest another penny in IMS?

Voice telephony needs priority, but after that all people want is a dumb pipe. In a competitive market that what they'll get. So IMS is vastly more complex than appropriate.

The cost effective solution is simple (i.e. two) priorities on the radio link, softswitch-controlled circuits-over-packets for voice telephony and straight off-load to the public Internet for everything else. An IP/MPLS core network is more than sufficient.  And existing GSM voice telephony protocols are both adequate and appropriate for voice telephony.  Existing GSM signaling is already being carried over Sigtran over IP, so the pieces are falling in place.

What I've just described matches what I've seen of China Mobile's new 3G backbone.  If true, that means the Chinese network equipment providers (Huawei, ZTE, etc.) are well positioned to win 3G business in high growth emerging markets where costs are key.

It certainly sounds bad for IMS.

NGN ≠ the Internet, and never will

I see and hear a lot of confusion about next generation networks (NGN).  In most cases people are using the term roughly as the ITU-T defines it:

A Next Generation Network (NGN) is a packet-based network able to provide services including Telecommunication Services and able to make use of multiple broadband, QoS-enabled transport technologies and in which service-related functions are independent from underlying transport-related technologies.

but many people don't realize how little this has to do with the Internet.

The Internet is a "network of networks" that includes millions of smaller domestic, academic, business, and government networks interconnected using IP.  It is a hierarchy because there is a backbone of ~28,000 autonomous systems (ASs) which exchange IP packets using routes established by Border Gateway Protocol (BGP).  The remaining millions of networks connect to that backbone via hundreds of thousands of ISPs and other intermediaries who are ASs or connect to an AS.

All of the NGN proposals (Wikipedia has a good summary) involve sophisticated QoS.  But it is well established that there is no technical or commercial requirement for QoS on the Internet backbone (references discussed here and here).

The thousands of organizations that are ASs exist in hundreds of different jurisdictions.  While some ASs are heavily controlled by governments (there is basically one AS for all of China), AS interconnection is independent of any single government.  Interconnections occur based on tradeoffs between the cost of doing business locally and the cost of routes to other locations.

Indeed, to the extent Tier 1 ISPs have attempted to limit free peering, Tier 2 ISPs have established peering agreements that form a donut around the Tier 1s, thus cutting Tier 2s' transit costs to an absolute minimum.  So the effectively unregulated Internet backbone is working remarkably well based on commercial arrangements between thousands of parties, just as it has for 15+ years.

With no technical need for QoS on backbone routes (as discussed here) and no commercial reason that anyone has articulated, it's hard to see how the thousands of parties who make up the core of the Internet would agree to do anything with QoS, ever.

Established telephone companies will deploy NGNs for telephone service.  To the extent they have a monopoly on Internet access, they will be able to use their NGNs to block access to the Internet, but the existence of NGNs won't change the way the Internet core works or the way anyone else's network works.

So NGN's are an evolution path for existing telephony networks, not the Internet, and they will last as long as the existing telephone service model lasts and no longer.

Why there is no Internet QoS

In 2006, I wrote a short blog post on why there is no Internet QoS and likely never will be.  That post is a continuing source click throughs and email inquiries so, when Jon Arnold asked me to write a guest article for IP Convergence TV, I thought I'd tackle the definitive story on Internet QoS.

Here is the new article: Why there is no Internet QoS.  It may or may not be definitive, but I have filled in the engineering and economic background, and I investigate where QoS has been deployed in networks in order to identify where there may be QoS related opportunities for ISPs.

My conclusions:

• There is no Internet QoS today, and it’s unlikely any complex QoS scheme will ever be added to the Internet as a whole.
  
• To the extent next generation networks with rich QoS are deployed, this will only happen within walled gardens, not as part of the public Internet.
  
• But there may be opportunities for ISPs to create a simple premium service that could generate incremental revenue.

Tracing Internet connections by inter-packet timing correlation

Susan Crawford's post The Auction, the Cops, and Comcast, highlights a request from the US Department of Justice to the FCC that is likely to result in an FCC Notice of Proposed Rule Making (NPRM).  As I read her post, I recalled an interesting paper on surveillance.  Here's the relevant part of Susan's post (bold italic highlighting is mine):

The NPRM, if it follows the DOJ’s request, will suggest (among other things) that all of these providers should build their routers and network hardware to provide “packet activity reporting” for all packets crossing their networks, and physical location information for all of their customers at all times. It will also suggest that very fine-grained timing information is needed - something that the internet and its applications don’t provide at the moment. “Packet activity reporting” means that the broadband provider will need to know the destination IP address and port number for everything happening on its network.

The idea is that these designs will help law enforcement when they want to carry out a request for call-identifying information.

A few months ago I happen to read a 2002 paper by Xinyuan Wang, Douglas S. Reeves and S. Felix Wu entitled Inter-Packet Delay Based Correlation for Tracing Encrypted Connections Through Stepping Stones.

I should comment that, when I travel in China, I use TOR (The Onion Router)
to bypass the Great China Firewall.  TOR works by passing your traffic through a series of intermediate routers, with intermediate connections encrypted, so a third party observer can't tell whose traffic is going where.  This is mildly useful to me when I'm in China, but it can be life saving for dissident writers living in totalitarian states. 

So it was somewhat distressing when I read the paper by Wang et al.

From their abstract:

... we address the problem of tracing encrypted connections through stepping stones. The incoming and outgoing connections through a stepping stone must be correlated to accomplish this. We propose a novel correlation scheme based on inter-packet timing characteristics of both encrypted and unencrypted connections. We show that (after some filtering) inter-packet delays (IPDs) of both encrypted and unencrypted, interactive connections are preserved across many router hops and stepping stones.

and from their conclusion:

Our correlation metric does not require clock synchronization, and allows correlation of measurements taken at widely scattered points. Our method also requires only small packet sequences (on the order of a few dozen packets) for correlation. We have found that after some filtering, IPDs (Inter-Packet Delay) of both encrypted and unencrypted, interactive connections are largely preserved across many hops stepping-stones. We have demonstrated that both encrypted and unencrypted, interactive connections can be effectively correlated and differentiated based on IPD characteristics.

So it's clear what the Department of Justice has in mind.  Of course, if the Department of Justice gets this through, it will just be an added expense on all ISPs (and thus on their customers, i.e. you and me).  It won't actually work against the bad guys (or the good guys) as it's fairly simple to imagine an outbound packet scheduler that introduces jitter into each flow at each onion router.

Singapore's Next Generation National Broadband Network

My post on Tuesday was in response to a Singapore government press release that used the phrases "structural separation" and "passive network." 

Subsequently I've been pointed to this website where the Singapore government has posted the qualification documents, mainly document IDA(PQ)-010V2, dated 11 December 2007.  It appears the actual RFP is only available to qualified bidders, but the qualification document has good information on the project structure.

Here is their view of structural separation:

Here is the supporting text:

3.3.2  There will be three distinct types of functional entities operating on the NGNBN.  At the lowest layer, the NetCo would be responsible for the design, build and operation of the NGNBN's passive infrastructure, which includes ducts and wirelines.  The NetCo would offer this passive infrastructure at a wholesale, non-discriminatory and cost-competitive basis (Layer 1 Open Access) to the OpCo(s).  The NetCo's passive infrastructure would be rolled out extensively throughout Singapore.  The NetCo infrastructure is expected to be capital-intensive with massive implementation effort.

To me that implies multiple OpCos can get access to dark fiber.  Hopefully private companies, who want their own dark fiber from one building to another, would also be able to lease dark fiber.  However, they don't make this clear so we'll have to wait and see. 

Only the NetCo portion is covered by the current RFP.  The Singapore government expects the NetCo piece to be built and operated by a private company or consortium, and they are prepared to provide up to S$750M (US$521M) in grants to make this happen.  Responses are due 25 March 2008 and will be evaluated based on:

• Attractiveness of business plan to industry
• Quality of network infrastructure
• Level of Government grant
• Financial proposition and strength of bidder

This is one to watch!

Rate of growth of traffic at the Amsterdam Internet Exchange

Here's a great historical graph, from page 28 here, showing the growth in traffic at the Amsterdam Internet Exchange.

By my take off from this graph, there was 1000x growth in nine years.  That means traffic doubles every 11 months.  While this is only one data point, it's new to me and fairly up-to-date.  It's also roughly consistent with the 12 month doubling rate mentioned in my earlier post.

Managed Storage Futures

Recently I wrote about differences in the exponential growth rates of computing and networking and promised to say more about how these differences cause substantial shifts in the technology landscape.  Managed storage is one example.  The relevant doubling rates (from that earlier post) are:

Doubling Rates

Technology Measure	Months
Computing performance	18
Storage capacity	12
Networking performance	15
Access connectivity	20-26

The increase in storage capacity per dollar has been phenomenal and is one of the reasons that Google can offer Gbytes of free storage for email and that Amazon can offer their Simple Storage Service (Amazon S3) at extremely low rates.

But it's also caused headaches for IT directors, as installed equipment becomes obsolete long before it's fully depreciated, and employees and department heads grip about inflated internal billing rates for storage.  Pity the IT staffer who sends a broadcast message justifying corporate email storage limits because "it costs the company X cents per megabyte per month."  I've seen such messages, and the employee ridicule they engender.

This sounds like a perfect opportunity for a managed service — provide an interface that looks a storage area network or network attached storage, using multiple (for reliability and arbitrage) Internet-based storage services to provide the actual storage.  But now differential growth rates become a factor.

Storage costs decline a bit more rapidly than the cost of Internet transit.  So, already it's the case that network-based storage is extremely low cost for backup but less affordable for transactions.

But the real problem is the cost of access connectivity.  If you're selling managed services to IT departments, you need to provide services at their premises.  Local connectivity is not fast, cheap or reliable, and the pace at which it improves is glacial in comparison with storage or Internet transit.

Has this prevented the emergence of managed storage solutions?  Of course not.  But most existing solutions focus on remotely managing equipment that's physically on the enterprise premises.

Is there opportunity for network-based managed services.  Also, yes.  But you will need considerable focus on local connectivity, both for the numbers you use in your business plan and for the specifics of how you implement the service.  Some thoughts:  interface your managed service via a remotely managed on-premise box that includes caching?  use a dedicated access link to guarantee QoS?  ???

In any event, three years from now, you can count of disk storage being ~8X more affordable, Internet transit being perhaps 4x more affordable, but local connectivity only 2x or 2.5x.  Don't give up your great  business idea, but plan accordingly.

Communications' Exponential Growth Rates

Everyone in high tech is familiar with Moore's Law and most are aware that similar exponential growth occurs in many other measures of digital technology, for example disk storage.  What about communications?

Gordon Moore referred to the number of transistors that could be inexpensively placed on an integrated circuit.  The Wikipedia entry for Moore's Law mentions other measures that are increasing at exponential rates as alternate "formulations" of Moore's Law, e.g., computing power, hard disk capacity and pixels per dollar.  Unfortunately, Wikipedia (at least so far) doesn't have accurate doubling rates or supporting data for these other measures and they present little on the area that most interests me -— communications.

Why care?  Different rates of exponential growth can cause substantial shifts in the technology landscape, some of which I'll discuss in subsequent posts.

Here are the doubling rates for some measures I'm interested in:

Note 1:  Density at minimum cost per transistor, see the Wikipedia article on Moore's Law.

Note 2:  12-24 months depending on how computation is being measured.  See, for example, the arguments here.

Note 3:  Since the mid-1990s, disk capacity has doubled  every 12 months.  In earlier decades, it was more like 15 months (based on calculations from this data), but since 1995 (through today) the 12 month rule has held.

Note 4:  Based on these data points:

Note 5:  The classic reference is Odlyzko & Coffman (2001) (also 1998).

Note 6:  My personal Internet connectivity is documented here (for data to me).  For data rates from me to the Internet, the Feb 2007 value is 2.7 Mbps, which implies a 26 month doubling period (versus 20 months for data rates to me).

Off to Madrid for Connect 2007 Europe

The third and final Connect conference of 2007 is taking place in Madrid on Wednesday and Thursday, November 7th and 8th and I'll be there.  My blog comments on earlier conferences are here (& 1, 2, 3, 4, 5, 6, 7, 8).

Day One has a heavy focus on mobile industry issues and mobile applications. And, it's conducted as panel discussions with few or no slides.  Perhaps this only works because there are good speakers, chosen to promote controversy and discussion, but it really works!  In both 2006 and so far in 2007, the nature of the discussion has been much, much better than at a typical industry show.  The session descriptions for November 7th are here and the speaker bios are here.  If you can be in Madrid on Wednesday, you should attend.

Day Two is a more traditional developers conference focusing on NMS technology and products that are used to create many of the applications discussed by the Day One executives.  Check out the Day Two program.

The day the routers died (to music)

The RIPE 55 meeting in Amsterdam has ended and Raindeer just uploaded this summary to his blog Lunatic Thought:

"The RIPE 55 meeting has just concluded. There was much debate on what to do on the imminent depletion of the unallocated IPv4 pool in 2010. We could do nothing or we could create a market place and facilitate transfer of IP-adresses, but it's all a train wreck waiting to happen. This is best shown however by a beautiful song "The day the routers died" also available on Youtube written and performed by Gary Feldman. So please all upgrade to IPv6 soon, or else you will not get 40Gbit/s to your mother."

and points to this wonderfully funny video:

Internet Connectivity in Guilin, China — Great China Firewall, plus engineering & capacity issues

I'm in Guilin China for the Connect 2007 Asia conference.  I've got the Tor bundle (Tor & Privoxy & Vidalia) installed on my laptop so I can get around the Great China Firewall (GCF) and that works well.  Tor starts up automatically when my PC starts, but I only use it for browsing (by clicking the Tor button for Firefox) if I can't reach a site directly.  Yes, it's noticeably slower going through Tor, but it always works.  For example, just now, seeking the URLs to paste in about the GCF, I could get Rebecca MacKinnon's post directly, but timed out on this from Gigaom.  I clicked the Tor button, hit retry and the Gigaom post loaded.  So, the Tor bundle is a good setup which I highly recommend if you are traveling to China.

The bigger problem for me at the moment is basic throughput.  The Sheraton Hotel Guilin (where the conference is being held) has wired Internet connectivity in the rooms for 50 Yuan ($6.65) per day and free WiFi in the lobby, but only WiFi from China Mobile (CMCC) in the conference area (100 Yuan per day through the hotel business center for those of us without a China Mobile phone number).

Performance is roughly equivalent for all three connections.  Early in the morning or late at night (perhaps during the night, but I do have to sleep sometime), everything works well.  It's not the blazing speed I sometimes see in Beijing, but it works. Here are the results in my room at 7:37am against a server in Shenzhen (300 miles away):

and here's a test against a server in Los Angeles:

Unfortunately, by 10am things begin to slow down and by noon yesterday, I was seeing greater than 50% packet loss and it became impossible to blog the conference live.  I was using the China Mobile WiFi but going up to my room didn't help.

My guess:  There's a significant capacity problem between Guilin and the rest of China.  Not surprising I suppose, considering Guilin is a tourist city, not a business city.

IMS is about transport, not services

OK, that title is designed to grab your attention, but it's also the reasonable takeaway of people in the audience at the last session yesterday, i.e. day one at Connect 2007 in Boston.

The session was entitled "Increasing Service Velocity" and the session description focused on IMS and service delivery platforms.  I wasn't involved in setting up the session or signing up the speakers, but a few weeks ago I was recruited to moderate the session.  The panelists from left to right were:

• Kjell M. Johansson, Director of Solutions Management, Multimedia, Market Unit North America, Ericsson
• Susan Norris, Communications Industry Advisor, Norport Technology Management Consulting (but note that Susan spent many years at operators, most recently at Sprint, and was the most articulate about the operator point of view).
• Douglas Tucker, CTO NA, Ubiquity Software Corporation
• Jouni Welander, Head of New Solutions US, Nokia Siemens Networks

So there's a slight problem.  Everyone here is smart and knowledgeable, but everyone (with the possible exception of Susan) is involved in creating or selling IMS or IMS products.  Panel discussions are always better if you have divergent points of view, i.e. controversy on stage!  Since I was moderating, I obviously couldn't blog it live or even take notes.  Luckily, George Kontopidis did take notes (and the picture above), so that helps me reconstruct events.  George's complete notes are here: Brough1.jpg, Brough2.jpg and Brough3.jpg.

Some of the specific comments on service deployment platforms (SDPs), their relationship to the rest of IMS and to deployment of actual applications included:

Kjell - SDPs are essential to the development of actual services, but the problem here is too many standards in what's exposed to the developer.  Kjell alluded to pressures from operators that may result in the major equipment vendors converging their service creation environments, but he couldn't give specifics or a date (beyond soon, within the next year).

Susan said operators think of IMS as the solution for new services.  They are generally very conservative (particularly on the operations side) and wouldn't dream of opening up their networks for something like web services.

Doug offered that IMS and higher level development environments, including web services, are not conflicting.  IMS is the platform, but it's standardization stopped below the application layer.  IMS needs web services and/or other development layers to actually realize new services.

And it was Jouni who offered "IMS is not about killer apps, it's about a killer environment" i.e. IMS is the platform but it needs (the non-standard) SDP layer above to support service creation.

On the subject of what is actually real, there was some consistency.  Kjell and Jouni both said there were a few commercial deployments for specific applications like video sharing.  Susan said she couldn't identify any IMS deployments with full service, but knew of several with partial services, i.e. IMS lite.  And Doug commented that he knew of pieces of IMS deployed in many operators, but nothing that's pretty or matches the IMS vision as yet.  On the other hand, everyone on the panel was confident that things would continue to improve.  There were joking remarks that 2008 would be the year of IMS.  In response to the question of when I would be able to hand off a video sharing session across operators, there was some agreement that the GSMA was working on this, so it might be solved in the next 1-2 years (although it could take longer to propagate to AT&T !).

There's a lot more in George's notes...

Dean Bubley did the best job shaking up the panelists (via questions from the audience).

And the sense I was left with (as were several members of the audience with whom I talked later) is roughly as summarized in my title above.  IMS is plumbing that helps operators manage their networks and is a great platform to support a variety of new services, but it will take higher layers (not part of the standards) to actually facilitate new applications,  In addition, there are many, many other issues to resolve, both technical (like integration with billing systems and other operator IT infrastructure, simplification of handset diversity issues, and so on) and business model related, i.e the extent to which will operators open up to new applications.

In any event, an interesting session!

Architecturally induced denial of service susceptability

Fundamental design decisions have a big impact on how a network behaves under stress.  I just noticed two otherwise unrelated posts both touching on network design issues that result in (or protect against) network collapse under stress.

Yesterday, I wrote about August's Skype outage where the system took over 36 hours to recover from a collapse.  The issue was too many clients attempting to reconnect at once.  Instead of clients “backing off” when supernodes weren’t immediately available, they kept hammering away, trying to log in.

Last week I wrote about the ways the Internet and the Telecoms networks respond to congestion in the backbone.  In December 2006 an earthquake off the coast of Taiwan broke multiple fiber cables causing a congestion collapse on the Internet backbone in significant parts of Asia.  The issue was too many computers attempting to reestablish TCP sessions with the result there was no capacity left for any session to actually send data.

In both cases, there were externally induced problems but, rather than recovering, there was the equivalent of a denial of service attack, self inflicted!

Both of these failures have a fairly simple solution, at least architecturally.  Under conditions of severe overload, the system must be able to restrict new attempts (new TCP sessions, new Skype logins, etc.) to some small percentage of the available capacity.  This allows the rest of the capacity to serve the logins, sessions or calls that do get through, with the result that what capacity remains is put to good use.

As I commented last week, this is one place where the telecoms industry has the correct architecture.  When disaster strikes subjecting some part of the network to overload, it's easy to restrict new call attempts on trunks into the congested area, for example by call gapping. This limits the amount of new traffic to that which the network can handle.  Thus, if only 30% capacity is available, at least the network handles 30% of the calls, not 3% or zero.

Here's one place where network architects can learn from established telecoms practice.

Skype outage: post mortem evolves

Since my last summary, some new information has emerged.  Gerry Blackwell interviewed Skype's Director of Operations, Michael Jackson, got some comment from Martin Geddes, and wrote an interesting article.

I've also had an email exchange with Julian Cain, which I reprint below, and a brief exchange with Philippe Biondi of SecDev.org who referred my to an interesting blog post (in French) by his colleague Cédric Blancher at  EADS France, Innovation Works (Suresnes).

New information from Michael Jackson (in Gerry Blackwell's article) includes the idea of five supernodes per cell (of 300 users):

Each supernode handles about 300 nearby users. Skype configures five in each cell for redundancy. So with upwards of nine million users online, it takes something like 150,000 supernodes to make Skype work.

The triggering event is still attributed to massive computer reboots after Microsoft's Patch Tuesday.  Everything else in the article is consistent with the best earlier explanations, including Julian's which I summarized here, i.e. as Blackwell puts it:

... the real culprit, Skype now says—was a resource allocation algorithm in the client software that could not adapt to such a set of circumstances. Instead of clients “backing off” on their attempts to validate on the network when supernodes weren’t immediately available and waiting for the ship to right itself, they kept hammering away, trying to log in.

And the solution, having clients back off when supernodes aren't immediately responsive, is obvious.  What's left to understand?

1.  I still haven't seen a plausible explanation of why Microsoft's "Patch Tuesday" resulted in problems on Thursday morning, only a lot of questions.  If the problem was induced by massive reboots, why didn't it happen on Wednesday morning?

2.  I still haven't seen a reasonable discussion of scaling.  As I wrote back in August,

I wonder, apart from the login server cluster as a single point of failure, is there also a scaling issue?  FastTrack's breakthrough was the use of supernodes to make the system more scalable.  But was that just one layer of scalability?  If so, what happens when there are 300 million on-line users and one million supernodes?  Perhaps Julian (or another P2P expert) could comment...

Indeed I emailed Julian about scaling and also about Joost.  This was his reply.
On 09/05/2007 05:06 PM, Julian Cain wrote:

Brough,

Skype and Joost are utterly different however Skype is more like Fasttrack(Kazaa). Joosts' Network architecture is mainly "Centralized", they have their own server farm of Supernodes as well as Authentication and Jabber servers. The nature of Joost is less dependent on peer to peer routing as it's basis is tuned towards QoS. Joost peers route traffic and relay UDP based payload as media data streams as well as keep a small cache of what they have recently viewed, however currently every Joost peer is directly connected back to the Joost home servers unlike Skype and Kazaa where once authentication occurs it's "out of our hands".

I agree on the extent of Skype scalability being very limited because of the nature of the Supernodes. At any one time the Supernodes hold ~300-500 child nodes and maintain an "Overlay" network which consists of another several hundred Supernode to Supernode connections. Ie* The Supernode network is very dense in order to provide for best means routing of least cost, however the flaw in this architecture is where the "Overlay" network reaches a capacity and is unable to reliably route traffic. I do not currently have any statistics on how the "Overlay" layer is not scalable but as more Supernodes arise the management of the "Decentralized Data Store" becomes a very hard task as well as keeping this "Overlay" in one single "in sync" network. This was proven with the Skype outage as the network was "trying to heal" it had to start from many 10s of thousands of "Overlay" networks which very slowly were able to sync again as a "single" network however is still an issue today with presence.

For the current Skype "Overlay" network to scale indefinitely while maintaining  a "Single" network infrastructure it needs in place an organizational hierarchy of Supernodes and a level of Service for each of these Supernodes. *Ie. If Skype Supernodes worked in a way such as in Fasttrack then when the network reached 100 million users it would began to crawl. This is due to the dense nature of the upper "Overlay". I can only assume that Skype has thought of this and that when the Supernode ratio is beginning to "bottle neck" then there would be some ordered Hierarchy as to what role each Supernode was playing, otherwise the more Supernodes the more dense the "Overlay" the more the data is relayed back and forth before considering the "Supernode Overlay" into it's own Denial of Service attack.

I hope this helps to some degree, let me know if you have any other questions.

~Julian

So to the extent I have time to look into P2P technology further, I plan to explore what's been written about hierarchy in P2P networks.  Here are some references (which I've found but have not read as yet):

RFC 4981 on Survey of Research towards Robust Peer-to-Peer Networks: Search Methods

Hierarchical Peer-to-peer Systems by L. Garces-Erice, E.W. Biersack, P.A. Felber, K.W. Ross, and G. Urvoy-Keller.

An efficient peer-to-peer file sharing exploiting hierarchy and asymmetry, by G. Kwon and K. D. Ryu in the Proceedings of the 2003 Symposium on Applications and the Internet, 27-31 Jan. 2003 Page(s): 226 - 233.
< unfortunately only available on an IEEE pay-for site >

Congestion in the backbone: Telecom and Internet solutions

When a network is subject to a rapid increase in traffic perhaps combined with a rapid decrease in capacity (for example due to a fire or a natural disaster), there is a risk of congestion collapse.  In a congestion collapse, the remaining capacity is so overloaded with access attempts that virtually no traffic gets through.  In the case of telephony, everyone attempts to call their family and friends in a disaster area.  The long standing telephony approach is to restrict new call attempts upstream of the congested area, for example by call gapping.  This limits the amount of new traffic to that which the network can handle.  Thus, if only 30% capacity is available, at least the network handles 30% of the calls, not 3% or zero.

There are comparable issues for the Internet backbone, but they are not so completely solved.  As Wikipedia puts it:

Congestion collapse was identified as a possible problem as far back as 1984 (RFC 896). It was first observed on the early internet in October 1986, when the NSFnet phase-I backbone dropped three orders of magnitude from its capacity of 32 kbit/s to 40 bit/s, and continued until end nodes started implementing Van Jacobson's congestion control between 1987 and 1988.

TCP congestion control solved day-to-day congestion collapse, but it didn't deal with disasters such as the Taiwan earthquake of December 2006.

As a result of problems in the Internet backbone in Asia caused by that earthquake, there's new discussion of Internet congestion.  I stumbled on one interesting discussion on the NANOG (North American Network Operators Group) mail list.

In this thread, the NANOG group seems to be arriving at conclusions similar to those of the telecom industry, i.e. throttle new traffic attempts upstream of the congested area.  From Fred Baker:

So plan B would be to in some way rate limit the passage of TCP SYN/SYN-ACK and SCTP INIT in such a way that the hosed links remain fully utilized but sessions that have become established get acceptable service (maybe not great service, but they eventually complete without failing).

And from Sean Donelan:

This would be a useful plan B (or plan F - when things are really FUBARed), but I still think you need a way to signal it upstream 1 or 2 ASNs from the Extreme Congestion to be effective.

...what should the alternate queue plan B be?

Probably not fixed capacity numbers, but a distributed percentage across different upstreams. ...

  Session protocol start packets (TCP SYN/SYN-ACK, SCTP INIT, etc) 1% queue
  Everything else (UDP, ICMP, GRE, TCP ACK/FIN, etc) normal queue

And finally why only do this during extreme congestion?  Why not always do it?

The thread contains over 50 messages and doesn't reach a specific call-to-action (that I could detect), but it does show similar problems and similar potential solutions.

I would have thought there'd be a few old telecom folks on the NANOG list, but I guess not...

Best Skype outage post mortem so far

It's been more than ten days since the global Skype outage – time to reconsider what actually happened.  The most credible analysis is not from Skype, but from Julian Cain in a series of comments (here, here and here) that he made to a Gigom article about the outage (or see the single file in "References" below).  Julian is lead architect at Pando and, earlier, was head of Mac development for Kazaa at Sharmen Networks.  So he knows a lot about peer-to-peer networks and his work at Sharmen put him in a position to know quite a bit about the P2P technology that's also used by Skype (and likely by Joost).

Background

Skype's P2P technology was evolved from FastTrack, originally developed for Kazaa.  Their P2P network consists of clients and supernodes.  Skype distributes client software which includes all necessary supernode software, so any client that has appropriate capacity and connectivity can be promoted to become a supernode.  Supernodes dynamically link to other supernodes to support a distributed database and distributed index (called the distributed hash table or DHT).  For Skype, the DHT layer is responsible for maintaining client presence info, contacts and icons/avatars, and handling call routing.

Root Cause

But as I pointed out in several posts during the outage, there's also a centralized component to the Skype network.  That's the login servers.  Julian refers to them at the "authentication servers" and/or "login/connectivity servers."  They are implemented as one cluster of about 50 machines.  As for the root cause of the outage, he asserts:

Skype employees introduced code into the "login/connectivity" server farm that was not compatible with current Skype clients.

Other Issues

While that was the root cause, it was helped along by other network characteristics, notably that each client connects to only one supernode at a time.  According to Julian, there are 300+ clients per supernode and if a supernode goes off line, the 300 or so clients connected to it must reenter their "connecting" sequence, i.e., find and connect to another supernode.

A network with 8 million on-line users implies ~27K supernodes, a figure that's consistent with the ~20K supernodes estimated by Desclaux and Kortchinsky in 2005-2006 (see their June 2006 Recon presentation, PDF here).  The other point from measurements by Desclaux and Kortchinsky is that each supernode attempts to maintain a list of all other supernodes which means there is a substantial amount of traffic between supernodes.  This clearly contributed to the slow recovery, during which Julian commented:

Right now there are approximately 10,000 Skype networks instead of one single "in sync" network.

Scaling?

So I wonder, apart from the login server cluster as a single point of failure, is there also a scaling issue?  FastTrack's breakthrough was the use of supernodes to make the system more scalable.  But was that just one layer of scalability?  If so, what happens when there are 300 million on-line users and one million supernodes?  Perhaps Julian (or another P2P expert) could comment...

References:

I've extracted and assembled a complete copy of Julian's relevant comments.

Vanilla Skype part 1 and part 2, by Fabrice Desclaux and Kostya Kortchinsky, Recon, June 17th, 2006.

Skype traffic during the week of the outage, captured by Phil Wolff of Skype Journal.

Skype's centralized control of P2P network parameters

The blogsphere is abuzz with reaction to Skype's second attempt to explain what caused the recent crash of their entire "peer-to-peer" network, but I haven't seen any comment on the one thing that struck me (in their 4th paragraph):

Once we found the algorithmic fix to ensure continued operation in the face of high numbers of client reboots, the efforts focused squarely on stabilising the P2P core. The fix means that we’ve tuned Skype’s P2P core so that it can cope with simultaneous P2P network load and core size changes similar to those that occurred on August 16.

As I commented earlier, we know from presentations by Desclaux & Kortchinsky at Blackhat Europe (PDF) in March 2006 and at Recon in June 2006 (PDF in 2 files: one and two), that there is substantial traffic between the (3rd-party-owned, distributed, P2P) supernodes that form the core of the Skype P2P network and Skype's (centralized) login servers.

If Skype's explanation is correct, it's clear Skype also has a way of distributing parameters to supernodes that tune their behavior.  I'm not surprised.  It's a logical to design in both measurement and tuning capabilities. 

But such centralized capabilities also represent a potential venerability.  What would happen if a black hat got access to those tuning capabilities...

Understanding the Skype Outage

Skype's official explanation.  Phil Wolff has a good set of interpolated comments on the official explanation.  There are two things to add.

1.  As the Register points out, last Tuesday was Microsoft’s monthly patch day and those patches required a re-boot.  If we believe Skype that their problem started with excessive login attempts, this is the only plausible explanation on the table.

2.  There was no patch for the Skype client (i.e. this was routine and hasn't been widely adopted) so either:

• we face another problem next time Microsoft issues patches requiring a reboot, or
• they fixed something on their servers. 

I suggest the latter.  As I pointed out during the outage, Skype generates a lot of traffic between the login servers and supernodes (see slide 16 in  DESCLAUX and KORTCHINSKY's presentation.  I suggest Skype has patched something on the login servers.  It's well known (e.g. Desclaux & Kortchinsky) that Skype login is a centralized function.

Meanwhile, it will be interesting to see if any additional comments or new client releases appear from Skype in the coming days.  I suspect not, as their approach to security has always in included both encryption and obfuscation.

Skype crash probably not DOS

Skype says it's a software issue that they are working on.  For me, it's six hours now and I haven't had more than a few minutes of "connectivity" in the past hour.  And, when connected, the on-line count was at a new low:

Another glance through DESCLAUX and KORTCHINSKY part 2 suggests, in addition to traffic from clients to login servers, Skype generates a lot of traffic between the login servers and supernodes (see slide 16 for example).  This makes the login servers a doubly critical piece of Skype infrastructure.  Of course, if they merely introduced a bug in their login servers, it shouldn't take 12-24 hours to do a roll back.

Skype Network Crashes

As others are reporting, Skype clients have been disconnecting and reconnecting around the world.  Here is Boston, I've been off line and back again at least four times in the past hour.  And when I've reconnect, an amazing small number of others are seen as on-line:

In recent weeks, I've been seeing over 9 million users on line at this time of day, so 615K suggests very little of the global Skype network is accessibile to me, if they are on-line at all.  A few minutes ago, Jan was seeing  773K other users from his site in Malaysia, so this really is global.

We've known, at least since 2004, that Skype's peer-to-peer network wasn't strictly P2P.  The vast majority of traffic (control and media) is P2P, but everytime a client comes on line (well at least at startup and each login), it interrogates the Skype Login Server at skype.com.  We also know that a Skype client must establish a connection to a super node to successfully login.  But I don't know if there is anything about supernodes that cause them to crash if they can't reach a centralized or semi-centralized Skype server.

It will be interesting to see how this develops.  Hopefully Skype will be forthcoming, but if not, I'm sure third parties will piece together an answer.

Latency and throughput using iPhone on EDGE

The following represents just one set of tests on just one evening at just one location in the Boston suburbs, but perhaps someone will find it useful.

Thanks to Jacob Barss-Bailey who made the measurements (on August 1st) and gave me permission to reprint them.  From Jacob's email:

The latency I was able to see last night, which was likely on an unloaded network (it was quite late, and I'm out in the country) was between 500ms and 2500ms, about what I was expecting.  About 75% of the time, I was seeing latency between 500ms and 700ms though, so it seems that latency is usually pretty good, but when its not, its pretty bad.

FYI the bandwidth I was seeing downloading random data was between 60kb/s and 120kb/s, heavily skewed towards the top of the range.

Obviously, as it's an iPhone, these are measurements on AT&T's EDGE network.

Well it's consistently better than dial up...

IMS deployments piecemeal until Release 7

Recently I was interviewed by James Tulloch of IMS Vision who proceeded to write this article quoting me extensively.  Very interesting (of course!).

My earlier comments on IMS realities are here and here.

VON: more on AT&T's new network plan

Yesterday I posted a gut reaction to Siroos Afshar's talk on AT&T's new network plans.  Looking at the rest of my notes and surfing the (real) Internet today, I find there are zero hits for 'Common Architecture for Real-Time   Services' or 'AT&T CARTS' i.e. the name and acronym he used for their planned network.  So public presentation of CARTS must be relatively new!

Indeed, inquiry among some friends suggests there is a 180-page specification being circulated to major equipment providers under NDA.  I haven't seen it, so I can't give first party confirmation.  Also, while many VON presentations are available to registered conference attendees here, Afshar's is not.  Perhaps the little additional information in this post will be of use to someone.

CARTS seeks to decouple services from the various access networks and position their middleware to handle service logic, for all their services, using network databases of user preferences (including complete description of user devices) and user availability (presence).  His point about IMS was that it handled part of their requirements (centered on connection control) but was incomplete for AT&T's purposes.  He had a slide with a list of all the extra capabilities they needed, beyond IMS.  I wish I had transcribed that one (or photographed it), but no such luck.  I'll keep checking the PulverMedia site in case his presentation appears, but since it was on a Pulver server at the show yesterday and it's not already available to attendees, I'm not hopeful.

VON: Siroos Afshar on AT&T's view of IMS

This talk is the best example I’ve seen yet of the total disconnect between the “Internet” crowd and today’s communications operators!

The opening session at Spring VON this morning was “Communications Networks of the Future and the Role of IMS” by Siroos Afshar, Network Architect, AT&T.  Siroos provided a view completely at odds with any of the Internet folk I know.  Indeed, he started by warning that by ‘network’ he does not mean the ‘Internet’ but rather the network of an operator who is offering services.

He said his first diagram was so high level there should be no controversy, and indeed, it has users, user devices, access technology, service middleware and service.  And yet it doesn’t mention “Internet Access” as a service I might want!!!  He does have a circle labeled “Your App.”  Later he says the Internet is just another access technology to reach his Network.

In fact, it appears in later diagrams that his 'Internet access' means Cable or DSL networks that reach his middleware, not the ability to send IP packets to any other public IP address in the world.  His functional units diagram includes “Peering with other networks” but only talks about IMS and VoIP peering, not IP transit.  This is incredible!  The words that go with this diagram are focused on services delivered by AT&T, not including access to what I know as the Internet.

IMS facts of interest (at least to me): 

• He’s focused on IMS release 7 which makes sense, as release 5 was incomplete and probably would never have worked. Also IMS release 7 is the first version that actually merges the work of TISPAN with that of 3GPP.
• He recognizes that IMS only handles connection control and gives a large list (which I wasn’t able to copy in time) of other capabilities that AT&T requires.  At a minimum, he calls out service creation environments as a big issue and, even larger, the operations, management and billing infrastructure which is needed beyond IMS.

The second question in the Q&A asked, “what if I want a service from somewhere else on the Internet?”  His answer, “If you want to access other services elsewhere on the Internet, that’s fine.  I was only talking about how AT&T plans to deliver AT&T services.”

Fair enough, but it was a surreal experience to hear this particular talk at VON.

 

At VON Next Week

Next week I'll be attending and speaking at Spring VON 2007 in San Jose.  If you'd like to connect, send me an email (rbt@nmss.com) or IM me on Skype (brough).  I'm arriving Monday afternoon and leaving Thursday mid-day.

I'm speaking on Tuesday afternoon (3:15pm) on the panel, Lessons Learned in Implementing IMS.  While our implementation is not complete, I'll be talking about what we've learned so far as we adapt our MyCaller Ringback Tone service to fit an IMS environment.

NMS is also exhibiting at VON (Booth 1019) focusing on our developer products including:

• Vision Media Gateway with Signaling, a complete, scalable, highly available solution for connecting IP-based value-added services to legacy PSTN networks, just being introduced at VON,
• Vision VoiceXML Server, unique as an integrated video and voice server for rapid deployment of IVR and IVVR systems,
• AdvancedTCA media solutions, i.e. our MG 7000A media blade.

In addition, we have a pod devoted to video development platforms as video is a big part of this show and NMS has a significant presence in mobile video.  Among other things, I'm hoping we'll be able to approximate the Mobile TV demo we routinely run in Europe and Asia based on the live service on the Hong Kong CSL network.  In Barcelona last month we were directly calling a backdoor number in Hong Kong (I still haven't seen the roaming bills for that week!).  Unfortunately in the US, the only 3GSM, i.e. UMTS, network is Cingular's and, so far, Cingular is the first and only 3GSM network in the world to roll out 3G without support for 3G-324M mobile video telephony.  So if we get the demo running in time, it will be running on mobile handsets using WiFi.

It's sad that the US is so far behind the rest of the world in mobile features, especially as we lead in minutes of use (i.e. talking)!

See you in San Jose next week.

Dealing with IMS —€” What an App Developer needs to know

As promised, the slides and recording of last Tuesday's webinar are up on the NMS website. 
Some of the key points: 

• There are strong business and service parallels between the Intelligent Network (IN) and the IP Multimedia Systems (IMS).
• IMS is only being used for new applications, and only a few at that, notably Push to Talk, Video Sharing, Fixed-Mobile Convergence and perhaps, Instant Messaging (although so far that's mostly talk).
• While there's work involved in integrating an application into an IMS network, it's no more complicated than integrating with IN — indeed, the larger issues are usually with OSS/BSS/NMS integration and they are relatively unaffected by IMS.
• Although it would appear that standard IMS infrastructure includes all the necessary components for application control (the Application Server), subscriber data (the Home Subscriber Server) and media processing (the Media Resource Function), most substantial applications end up shipping with their own additional application processors, databases and media resources.  These additional application-specific components need to connect via IMS standard interfaces, but need not (and generally do not) conform to IMS specifications internally.

Based on questions during the webinar, and by email afterward, I've hit on some of the practical issues developers are running into as they migrate existing applications to run in an IMS network.

3GSM Wrapup — Tons of opportunities & growth, but still focused on the walled garden

It’s been a busy week.  I had intended to post every day, but the level of activity was overwhelming, and minimum sleep flying to Europe followed by five hours of sleep per night while there just got to me.

In a business sense, this was a very good show, for NMS and seemingly for many others.  There’s tons of opportunity at many levels.

Taking the longer view, I worry.  Tim Berners-Lee, in his keynote address, beseeched the industry to “be a foundation, not a ceiling.”  He talked of the enormous opportunities that the open Internet has generating.  The walled gardens of the mobile industry are stifling this potential.

Of course they’re going to collapse.  Countries with mobile oligopolies may preserve their walled garden for years but, as serious 3G bandwidth and other sources of connectivity (WiMAX, WiFi) roll out in developed countries with real competition (3+ competitors making significant investments), the walls will collapse.  I’m talking 2–5 years.  Based just on anecdotal conversations, I think many of the operators realize this, but their whole structure and organization is such that all they can do is milk the current situation, leaving their future to be dealt with in the future.

Highlights:  Mobile communities, Mobile TV (& here), talk about mobile advertising (most of which I missed), Femtocells (also here), and WiMAX (which grates on 3GSM types), but not much progress on GSMA instant messaging (last year’s buzz).  Also, last year’s radio hype was HSDPA, so I expected hype for HSUPA this year, but evidently it’s not quite ready (or it’s too little and will be too late to counter WiMAX?). 

In any event the radio technology hype was diffuse, but included multiple announcements about 3GSM’s Long Term Evolution (LTE) from DoCoMo, Ericsson (and others I missed).  This is an obvious counter to the WiMAX crowd, but it’s a lame response, as LTE demos are a few years off with deployment still further.

Finally, the show was even bigger than last year — claims are between 57K & 60K people attended.

Here’s the NMS booth on opening day.

What was in it?  MyCaller ringback tones, of course.  We now have 27 operators who’ve deploy our system.  But we showed several new applications for the first time — Interactive 3G Mobile TV and Mobile Publishing, both based on 3G video calling technology.  And on the developer front we showed the first AdvancedTCA media blade, our MG 7000A and the latest release of our Vision VXML Server targeted at IVVR (interactive voice and video response). 

Our strategy is to foster a rich developer community and then partner with developers in that community to bring new applications to our operator footprint.  Thus the Vision VXML Server is the core of various IVVR (interactive voice and video response) platforms, including those that run Interactive 3G Mobile TV and Mobile Publishing.

I’m writing this on the plane (using BlogJet software) but I’m fading, so good night for now.

DoCoMo uses MIMO to achieve 50 bps/Hz

That's 50 bits per second per Hertz of radio spectrum.

Under the title NTT DoCoMo Achieves World's First 5Gbps Packet Transmission in 4G Field Experiment DoCoMo announced a new level of modulation efficiency.  Their title is flashy, but they were using 100MHz of spectrum.  To find that much spectrum you have to look to very high frequencies and thus short ranges with poor penetration of buildings (or trees and shrubs). 

The benchmark to follow is modulation efficiency — how many bps can you cram through a limited amount of spectrum.  Today commercially deployed systems tend to achieve less than 10 bps.  Here's an excerpt from a picture of their setup.

The full image with explanation is in this pdf file.

In some sense, this is a brute force expansion of earlier work with an increased number of antennas at slightly different positions being used to send different information on the same frequency simultaneously and multiple receiving antennas used to resolve the individual transmitted signals, i.e. multiple inputs and multiple outputs (MIMO).  It also builds on the most current modulation scheme which is orthogonal frequency division multiplexing (OFDM), the modulation used in WiMAX.  None-the-less, it's an impressive achievement.  Thank you DoCoMo.

Everyone has heard of Moore's law describing the decades-long continuous exponential increase in performance of integrated circuit technology.  Fewer people comment on the continuous exponential increase in radio spectrum utilization that's been in progress for more than 100 years, i.e. since Marconi patented "resonant tuning" in 1900, showing how he could send multiple signals over the air at the same time (versus a single transmission in his initial system of 1896).

Mobile VoIP — When will it take off?

I've started writing a monthly column for Internet Telephony Magazine. My first article examines when and how VoIP will actually happen in the mobile space. 

Mobile telephony continues to be the fastest growing part of the telephony market, far exceeding VoIP in absolute numbers and in the scale of new subscribers and new revenue added per year.  But mobile voice telephony is nearly 100% based on circuit switching.  When will this change?  For answers, see my column in the magazine or on the web.

Interestingly, Internet Telephony Magazine doesn't offer RSS feeds for the electronic versions of their magazines, so there is no way to subscribe, on-line, to my monthly column.  I've bugged Rich Tehrani and Zippy Grigonis about this.  Feel free to add your comments.   :-)

Mobile phones vs. PCs to access the Internet - Crossover in 2007-08

As I catch up on back reading, I came on Tomi Ahonen's "Putting 2.7 billion in context: Mobile phone users".  It's a long but interesting article, worth printing out and reading in detail (something I can't do in the airport lounge) as it puts mobile phones in context:

...context. 800 million cars, 850 million personal computers, 1.3 B fixed landline phones, 1.4 billion credit cards, 1.5 billion TV sets. How many mobile phones in use today? In use today, yes, 2.7 billion (technically 2.7 billion in January, not December). They sold 950 million phones last year and the total worldwide mobile subscriber base grew from 2.1 billion to 2.7 billion. Three times as many mobile phones as automobiles or personal computers. About twice as many mobile phone owners as those of fixed landline phones or credit cards. And almost twice as many mobile phones in use as TV sets.

Yes, that 2.7 billion subscriptions includes duplicates.  There may be 2.4 to 2.5 billion discrete users, but the context is clear.  More importantly for the Web 2.0 crowd to think about:

During 2007 the first cross-over will happen, with more users accessing via phone than PC. Fascinating data coming on that usage as well, the Japanese regulator reports that those who access the web via mobile phone do so more frequently than those who access via a PC. Similar data now coming from several converged (broadband and mobile phone) web services like Flirtomatic in the UK. Its no wonder Google's new CEO Eric Schmidt, says the future of the internet is mobile.

So pay attention to Internet and mobile phone usage in Japan, Korea and China as there, mobile phones are already the dominate form of Internet access.

Asian submarine fiber route detail

James Seng has an excellent illustrated post showing submarine fiber route in Asia and providing an expert explanation of the impact of last Tuesday's magnitude 7.2 earthquake off the coast of Taiwan.

If you are at all interested in submarine fiber, the Internet backbone or the Internet in Asia, this is the article to read.  It beats anything else I've seen in the press.