Communications

Limitations of Carrier Grade NAT, and some workarounds

Qtel, the largest carrier in Qatar (and nearly the only Internet provider) appears to connect all their users (~600K) to the Internet through just one or a very few public IPv4 addresses.  82.148.97.69 was their single public address in 2006-2007.  How can network address translation (NAT) put all those users through just one IP address?

Well, both UDP and TCP include 16-bit port numbers, so for each IP address there are 2**16 = 65,536 unique IP address/ TCP port number combinations which can support network address translation (NAT) for more than 65K simultaneous connections.  Of course, individual browser sessions may open multiple connections so each on-line user can tie up several such connections, even several dozen.

But there are more unique fields that come into play.  On the Internet side of the NAT, each active IP session has five fields of interest:  source IP address, source port, protocol, destination address; destination port.  So you can use the same source address and port number for multiple independent sessions if the sessions are going to different destinations or using different protocols. With this approach, one public IP address can clearly support millions and millions of connections.

But what if most of your users are trying to reach just a few destination addresses?  That’s a problem Qtel had (or at least their users had) when accessing Wikipedia in 2006.

One obvious solution is to devote more than one public IP address for the whole country!  Indeed, if a few hundred people are attempting to access the same address simultaneously, then having a few hundred IP addresses (for the whole country!) would do it. Of course, if the destination IP address is that of Google’s search bar, you might need more than a few hundred IP addresses.

There is another path, one which Geoff Huston alludes to in a recent article on the IPv6 transition.

… we may expect to see a push to re-architect content into Content Distribution Networks that have points of presence in major access networks.

Today, major content distribution networks (CDNs) like Akamai, Amazon Cloudfront, Limelight and Google each maintain thousands of servers close to the edges of the Internet.  These servers cache popular content close to where the users are.

When many, many users in Qatar simultaneously try to access the same content, the load on Qtel’s NAT boxes could be dramatically reduced by putting the appropriate CDN’s caching servers within Qtel’s private address space.  Then requests for popular content wouldn’t even go through the NAT, reducing total traffic and dramatically reducing the occasions when two users are going through the NAT to the same public IP address.

It’s not in line with the Internet’s original end-to-end design, but as IPv4 addresses get more expensive, we can expect to see more and more use of NAT and CDN servers moving into carriers' private address domains.

 

Comparative broadband speeds - US loses again

Recently Google Labs added the Ookla Speedtest data set to their wonderful Public Data Explorer so I just had to try it out.

Here's my first graph:

These are not bogus statistics. These graphs show the average of all the millions of actual speedtests run in the respective countries over the past 4 months.

The apologists for the US's poor showing usually start by pointing out that the US has much lower population density than South Korea, Sweden or Bulgaria. Of course that doesn't hold water if you compare Seoul, Stockholm and Sofia to New York, Los Angeles or Chicago. In Seoul, Stockholm and Sofia consumers can get 100/100 Mbps Internet access at affordable prices (e.g. $13/month in Stockholm) while such service isn't even offered to consumers in New York, Los Angeles or Chicago.

 

Paid Peering: Issues and misunderstandings

Recently I was asked my opinion of Google paying France Telecom (FT) to deliver traffic to into FT's network, i.e. Google paying to peer with FT. I wasn't aware Google pays FT. I don't even know if it's true. But I do know this is a topic fraught with misunderstandings. Also, if there is a "problem" here, the problem is one of competition (or lack thereof) in portions of the French broadband access market. It is not a problem that can be or should be fixed by "network neutrality" regulations or legislation.

To understand what's happening, you need to understand the Internet's business model and you need to know something about peering as it works in the real world.

The most suscinct definition of the Internet I'm aware of is: 
Internet (n.) A voluntary agreement among network operators to exchange traffic for their mutual benefit.

This is very different from the telephone networks (fixed or mobile) where relations between operators are governed by laws and regulations, change at a regulatory pace, i.e. decades, and are continuously gamed by political processes.

Each Internet operator needs to be able to reach all possible Internet addresses, so interconnection is essential. If you are a small operator, like netBlazr, you purchase "Internet transit" services from a larger operator. Rates are volume dependent and arrangements are similar to those of any other business or individual purchasing an Internet access service.  If you are a large operator, a large content provider or a very large business, then interconnection costs and arrangements become negotiable. Two operators in Holland will likely agree to exchange that portion of their respective traffic that is destined for each other's networks at the AMS-IX which is local, rather than pay for Internet transit services that would likely carry that traffic to London or the US where a higher tier operator would handle the exchange.  Thus the two Dutch operators engage in peering for that portion of their traffic for which it makes sense.

This pattern is repeated throughout the Internet as each operator evaluates where, how, and with whom it makes sense to exchange portions of their traffic so as to minimize costs, increase performance or achieve other goals important to that operator. Thus there are vastly more interconnection contracts, formed under more diverse terms, for the Internet than ever was the case for the telephone network. Further, this network of agreements is in constant flux as individual operators find better deals.

It's instructive to follow the growth of YouTube from their founding through their acquisition by Google. When YouTube began, they had to pay standard rates for Internet access. As they grew to be a significant content source, their cost of Internet access fell as more and more operators asked to peer with them, as I discussed two years ago.

So between Google and France Telecom, who has negotiating leverage? and why?

France Telecom already has peering agreements with multiple other Internet backbones, so their customers have several paths to get to Google content.  Why would either party want to connect directly?

First, it might cut costs. Whatever pre-agreement path the traffic takes involves some cost. Perhaps that's money paid to other operators or it may just be the cost of maintaining the facilities needed for the pre-agreement traffic (e.g. to go via London).  I can't guess the specifics for Google or for France Telecom, but each has a possible interest here.

Second, it will likely improve performance. Even shaving 50 milliseconds off of round trip times (RTT) can improve user experience as accessing a single web page can require the exchange of dozens of messages, thus 50 ms less RTT can shave more than a second off the page load time. In a competitive market, France Telecom might want to advertise their network as the "fastest," but local access competition is not that great in France (or in many markets, like the US).  France Telecom has nearly 50% market share.  They are losing ground to Free, but apparently it's not enough to justify a marketing campaign based on the "fastest" network.  Google on the other hand, has a strong reason to want better user experiences. In Google experiments with search latency, longer latencies reduce the number of searches a user does per day. Given the number of Google search users on the France Telecom network, Google can directly compute their monetary return from cutting search latency by X ms. So Google's negotiators have hard numbers in their back pocket when they go to talk with France Telecom.

Is this good or bad? Well, if France Telecom doesn't have to compete on the performance of their service, that suggests a failure of competition policy. But then a solution should address competition in the French Internet access market, not government regulation of peering or the imposition of "network neutrality" regulations on operators that don't have a monopoly position.

For myself, I look forward to continued growth of new providers in France, like Free, as I discussed in the January issue of NGN Magazine.

 

Paul Baran and open spectrum

Paul Baran, an engineer who helped create the technical underpinnings for the Arpanet (the precursor to today’s Internet), died Saturday in Palo Alto, California.  While everyone else celebrates his early contributions to packet networking, I will remember him for his open spectrum advocacy.

I never had the privilege of hearing Paul Baran speak live, but as I became interested in wireless spectrum, one of the earliest references I encountered was this transcript of Paul Baran's Keynote Address at the 8th Annual Conference on Next Generation Networks Washington, DC, November 9, 1994.

His title says it all:

Visions of the 21st Century Communications: Is the Shortage of Radio Spectrum for Broadband Networks of the Future a Self Made Problem?

And here's an excerpt:

   The key point at issue that we will question is the widespread belief
   that we don't have enough radio spectrum to go around. This is a
   common, fundamental belief. Since we live in a world of scarcity or
   natural resources it is almost automatic that we believe that there is
   a shortage of frequencies. This particular resource is somewhat
   different.
   
   This morning, let's start by reviewing this presumption of a permanent
   shortage. Let's consider how, with an application of already known
   technology, we could create even a surplus of frequencies. What may be
   going on is an inadvertent shortage created by a regulatory structure
   which has yet to appreciate the potential capabilities of the new
   digital signal processing technology as applied to communications.

Paul Baran is hardly the originator of open spectrum ideas, but the complete transcript shows he once again led the industry.

 

Network Neutrality is the wrong fight!

Winning would mean giving up much more important rights – historical rights that were in place in the US as recently as 1995 and remain in place in most of Europe even today. 

We shouldn’t settle for network neutrality. It’s a poor substitute for what we had and much less than what we need. Let me explain. There are two topics to discuss. The first is “common carriage,” a centuries old legal concept that applied to the US telecom industry throughout most of the 20th century. The second involves communications protocols. Both topics are complex, so I will cover only what’s needed to understand why we shouldn’t accept network neutrality and why, at a minimum, we should fight for enforcement of existing common carriage rules. 

Network neutrality is about allowing any Internet application to run over an Internet connection, i.e. over a connection that uses Internet Protocol (IP). But under common carriage as it applied prior to the late 1990s, we had a more powerful right – the right to run any kind of network protocol, IP or otherwise, over a lower, simpler service which today we call a “bit stream*.” Why does this matter? Because real innovation is also possible at these lower layers and that innovation continues to be important. But today, such lower layer innovation is restricted to inside one building or one campus. Yes, we can tunnel some lower level innovations over IP, but not all of them and only at a cost. 

IP telephony (VoIP) is one place where problems arise. Most enterprises use IP PBXs internally, yet calls between enterprises use the PSTN. Many companies have attempted to address this gap, but progress is slow and expensive. Within an enterprise, IP telephony packets are given priority, but that priority is not supported on Internet access links and network neutrality doesn’t help. As a result, to interconnect VoIP calls, enterprises must lease separate dedicated access circuits – circuits usually based on bit stream access – to support “SIP trunks.” Up until the late 1990s, these circuits were regulated under common carriage. Today they are an unregulated monopoly, with prices derived from the cost of voice circuits 15-20 years ago, i.e. abnormally expensive for today. 

 Common carriage is the legal concept that, in exchange for government granted monopoly access to rights-of-way, the monopolist must carry anyone’s traffic over the resulting infrastructure, at regulated rates. For centuries this has applied, to canals, to roads, to railroads, to telegraph lines and, until nearly the end of the 20th century, to telecommunications lines. But during the legal battles after the Telecom Act of 1996, the FCC basically gave up on common carriage.

If we accept Network Neutrality instead of common carriage, we guarantee future innovations happen only above the IP layer. Innovation at lower layers will be restricted to enterprise or campus applications. That’s too bad as it was the existence of common carriage that allowed the Internet to develop in the first place. Do we want to eliminate that kind of innovation in the future? 

If anything, we should be fighting to extend the ideas of common carriage to lower layers, e.g. dark fiber. Installing dark fiber is expensive and requires access to rights-of-way that are limited. The installed fiber is capital expensive infrastructure that lasts for decades. Such conditions justify granting monopoly access, in exchange for common carriage and regulated rates of return. But when you light up a dark fiber, you use (relatively) low cost gear with a short life (even if it can survive for ten years, Moore’s Law renders it functionally obsolete within 2-3 years). What's more, there’s rapid innovation in opto-electronics gear. Just look at the order of magnitude difference in cost between enterprise and carrier fiber-optic gear. 

Today, the US is loosing leadership in all things Internet. Network Neutrality will just put a nail in our coffin. To stop our decline, fight for restoration of the common carriage principals that existed through most of the 20th century and still exist in law. To regain world leadership, fight to extent those principals to include access to dark fiber at regulated rates.

---------------------------------------------------------------------------------------

* Bit stream access.   In the 20th century two regulated services provided what the 21st century calls bit stream access. These were voice telephony and T1 circuits. T1 circuits directly carry a stream of digital bits. Modems allowed voice connections to carry digital bits, for example, for bulletin board services and other purposes long before the Internet became popular.

Stealth no more! Also, I need your vote

A few of you have commented on my various profiles that say I'm associated with Ashtonbrooke Corporation or with BigBroadband.Net.  Of course neither website provides much information...

Well our service launch is still weeks away and we haven't found a name we like yet, but we have entered the MassChallenge and that means we've revealed what we're up to. MassChallenge is running a competition for startups and they're giving out grants.  Since we are entirely self-funded so far, a grant would be sweet.

To see our entry you must register at the MassChallenge website, http://www.masschallenge.org/user/register?role=Supporter and then go to Competition, Team Pitches, search for "BigBroadband.Net" and click Details.

Once you are there, you can not only get an idea of what we're up to, but you can vote for us!  Ideally before midnight tonight.  :)

Please Vote for BigBroadband.Net

As a registered supporter of MassChallenge you can rank any team's entry.  Just click on the right most of the five stars under "Ranking" on the right of our details page to give us a 5 Star/Awesome rating, ideally by midnight tonight.

Thank you.

Longer term..., we're seeking a new name

Once we roll out our pilot network (in Boston later this summer), I'll have a lot more to say about how we expect to pull a wireless end run around the duopoly, at least in dense urban areas in the US.

Meanwhile, we are seeking a better name. WirelessEndRun (our internal name) is either brazen or obtuse, or both.  BigBroadband.Net says something about what we're doing but has a lot of negatives.

All suggestions gratefully welcomed.

Internet caching

This morning's mail included a query from a reader who'd been looking at my April 2009 post on Google's Peering and Caching Strategy.  His question:  Are ISPs allowed to transparently cache content?  Here's my answer:

The simple answer is yes. Caching is not only allowed, it is an essential part of some Internet functions (for example DNS, see below).  For a more complete answer, we must ask "allowed" by whom? At a minimum, there are technical, legal and commercial answers. 

 1. From a technical perspective, there are three reasons for caching. It reduces latency, it reduces traffic on upstream connections and it distributes load thus reducing load on originating servers. Today, latency reduction primarily benefits individuals browsing the web, while traffic reduction reduces upstream costs - an advantage for the ISP that may or may not be passed on to the consumer. But reducing latency and traffic have been important goals from the beginning of the Internet and both considerations have influenced protocol design along the way. For example, the Domain Name System (RFCs 882/883 superceded by 1034/1035) makes extensive use of caching to improve performance and reduce load on the root servers. And many compression protocols, like ROHC (RFC 3095), work by maintaining local state at the receiving end - effectively a mini-cache of earlier header fields. 

However, I assume you are focused on web caching. In the specific case of web caching, some relevant technical documents are RFCs 2186 and 2187. 

 2. From a legal perspective, some copyright proponents initially argued that caching violates copyrights. I am not aware of anywhere where such arguments have held up. In the US, the 1998 Digital Copyright Millennium Act specifically exempts ISPs from copyright liability for caching web content.

 3. From a commercial perspective, consumers of Internet connectivity (individuals, enterprises, government agencies, etc.) typically seek minimum latency access to any Internet endpoint using any legitimate protocol. In a competitive market, i.e. one with many ISPs, "legitimate" protocol means just about anything built on IPv4. In the future, that market definition may evolve to include just about anything built on IPv6. Of course, many people use only a subset of full Internet connectivity and thus choose an Internet access service that gives little more than http and email but costs less. 

Since latency and cost are key in any competitive market, web caching is widely deployed by ISPs. What's more, specialized content distribution services like Akamai have emerged to further reduce latency for specific kinds of content that might not be well handled by conventional caches. The interfaces to a service like Akamai's are based on Internet standards, but the internal workings are Akamai-specific. These sorts of services are not only "allowed," they have become a competitive requirement for many distributors of Internet content.

Human nature can't endure the current rate of change - from 1883!

I had an interesting discussion over dinner last night with a mother whose daughter is in college.  She was relating how her daughter follows the news, gets her opinions, etc.  Of course her daughter was blogging and using social networks to keep multiple lines of communications open at once.  This led to the typical lament, "the world is changing more rapidly than I can understand."  We hear this in many contexts.  Here's one that's fairly far afield for me, but still typical:

...technology is changing at a pace without precedence in human history. One day's marvel becomes a necessity of ordinary life the next. Rapid technological change permeates the whole of human existence, exhausting our mental ability to comprehend and cope."

By amazing coincidence, yesterday I also stumbled on this article from the Atlantic Journal written in 1883.

"The world is too big for us, too much is going, too many crimes, too much violence and excitement. Try as you will, you get behind in the race in spite of yourself. It's a constant strain to keep pace... and still, you lose ground. Science empties its discoveries on you so fast that you stagger beneath them in hopeless bewilderment. The political world is news seen rapidly, you're out of breath trying to keep pace with who's in and who's out. Everything is high pressure. Human nature can't endure much more."

Indeed, earlier technology transitions have also upset people -- think of the Luddites responding to the industrial revolution.  While the pace of change may be increasing, emotionally disturbing change has been affecting individuals for several centuries.

Customer owned fiber in Ottawa

Arstechnica had a nice article yesterday by Timothy Lee entitled The Long Tail following up on Derek Slater's article last week on the Google Public Policy Blog entitled What if you could own your Internet connection?  Both articles are about a pilot project in Ottawa.

The "tail" in Timothy's article is the "last mile" (or as I prefer, "first mile") fiber connection from individual homes to a network peering point or other aggregation point where individuals can then choose from among multiple competing ISPs.  The importance is, as Timothy Lee puts it,

Without the expense of rolling out last mile infrastructure to every home, many more ISPs could afford to serve a given neighborhood by running wiring to the peering point, leading to more competition and lower prices. Perhaps best of all, the growth of customer-owned fiber could make debates over "open access" and network neutrality moot, as robust telecom competition should prevent the worst of the monopolistic behavior exhibited by telco and cable incumbents.

Typical costs run below $3000 per household depending on density and the number of households that agree to participate.  Considering the cost of a home, this is one investment that should easily increase the value of the home, most likely by much more than the cost!

Mentioned by Derek although unmentioned by Timothy is the most important person here, Bill St. Arnaud, Chief Research Officer at CANARIE, a Canadian nonprofit that focuses on network infrastructure.

Bill is driving the Ottawa trial and is the leading proponent of condominium fiber as described in his presentations here.  His work has also had a major impact on my thinking about "layer zero" competition.

Either way, articles on the Google blog and at Arstechnica will help promote this excellent concept and, hopefully, improve the quality of discussion in policy circles.

Education due for radical disruption

The Internet has disrupted music publishing, encyclopedias, classified advertising, news and is poised to disrupt both television and the movies.  What else?

Education is overdue, but most of today's academics still don't get it.

For decades, even centuries, we've seen increased productivity in agriculture, in industry and most recently in services.  Productivity gains show up more useful output per person employed.  One place where this has not happened is education, as the graphs below illustrate.  Indeed, in education, schools compete on the basis of lower student faculty ratios, i.e. lower faculty productivity.

Individuals vary in how they best learn things but, in most cases, direct interaction with a teacher is a significant benefit.  So if the Internet is just about better access to information or remote video interactions with teachers, it may be wonderful, but it's hardly disruptive. What's needed is some way to multiply the number of teachers, dramatically.

A suggestion of how this might come about, i.e of how the Internet might disrupt the traditional academic process, comes from italki.com.  To quote Gang Lu,

italki is a language social network providing free language learning content. Unlike the traditional one-on-one education approach, italki is building an online community where any user can play two roles, tutor and learner. In italki, users can find language partners, post their foreign language questions which can be answered by other users, and join groups for language learning. The new version introduced a new feature called knowledge which is basically a Wiki service. Now users can not only share the language learning materials they uploaded but also collaborate on creating free language learning textbooks by their own. So italki is building probably the largest Wiki for language learning content!

This is not some academics opening their university course materials to the world (although I'm sure that's useful).  These are individuals contributing course materials and, more importantly, tutoring each other.  This approach tackles the hard problem, the student teacher ratio.

I'm sure there are other examples that I haven't stumbled on as yet.  In any event, it will be interesting to see how this sector evolves.  italki.com started in Shanghai and is already dealing with 16 languages and has global participation.