Skype outage: post mortem evolves

Since my last summary, some new information has emerged.  Gerry Blackwell interviewed Skype's Director of Operations, Michael Jackson, got some comment from Martin Geddes, and wrote an interesting article.

I've also had an email exchange with Julian Cain, which I reprint below, and a brief exchange with Philippe Biondi of SecDev.org who referred my to an interesting blog post (in French) by his colleague Cédric Blancher at  EADS France, Innovation Works (Suresnes).

New information from Michael Jackson (in Gerry Blackwell's article) includes the idea of five supernodes per cell (of 300 users):

Each supernode handles about 300 nearby users. Skype configures five in each cell for redundancy. So with upwards of nine million users online, it takes something like 150,000 supernodes to make Skype work.

The triggering event is still attributed to massive computer reboots after Microsoft's Patch Tuesday.  Everything else in the article is consistent with the best earlier explanations, including Julian's which I summarized here, i.e. as Blackwell puts it:

... the real culprit, Skype now says—was a resource allocation algorithm in the client software that could not adapt to such a set of circumstances. Instead of clients “backing off” on their attempts to validate on the network when supernodes weren’t immediately available and waiting for the ship to right itself, they kept hammering away, trying to log in.

And the solution, having clients back off when supernodes aren't immediately responsive, is obvious.  What's left to understand?

1.  I still haven't seen a plausible explanation of why Microsoft's "Patch Tuesday" resulted in problems on Thursday morning, only a lot of questions.  If the problem was induced by massive reboots, why didn't it happen on Wednesday morning?

2.  I still haven't seen a reasonable discussion of scaling.  As I wrote back in August,

I wonder, apart from the login server cluster as a single point of failure, is there also a scaling issue?  FastTrack's breakthrough was the use of supernodes to make the system more scalable.  But was that just one layer of scalability?  If so, what happens when there are 300 million on-line users and one million supernodes?  Perhaps Julian (or another P2P expert) could comment...

Indeed I emailed Julian about scaling and also about Joost.  This was his reply.
On 09/05/2007 05:06 PM, Julian Cain wrote:

Brough,

Skype and Joost are utterly different however Skype is more like Fasttrack(Kazaa). Joosts' Network architecture is mainly "Centralized", they have their own server farm of Supernodes as well as Authentication and Jabber servers. The nature of Joost is less dependent on peer to peer routing as it's basis is tuned towards QoS. Joost peers route traffic and relay UDP based payload as media data streams as well as keep a small cache of what they have recently viewed, however currently every Joost peer is directly connected back to the Joost home servers unlike Skype and Kazaa where once authentication occurs it's "out of our hands".

I agree on the extent of Skype scalability being very limited because of the nature of the Supernodes. At any one time the Supernodes hold ~300-500 child nodes and maintain an "Overlay" network which consists of another several hundred Supernode to Supernode connections. Ie* The Supernode network is very dense in order to provide for best means routing of least cost, however the flaw in this architecture is where the "Overlay" network reaches a capacity and is unable to reliably route traffic. I do not currently have any statistics on how the "Overlay" layer is not scalable but as more Supernodes arise the management of the "Decentralized Data Store" becomes a very hard task as well as keeping this "Overlay" in one single "in sync" network. This was proven with the Skype outage as the network was "trying to heal" it had to start from many 10s of thousands of "Overlay" networks which very slowly were able to sync again as a "single" network however is still an issue today with presence.

For the current Skype "Overlay" network to scale indefinitely while maintaining  a "Single" network infrastructure it needs in place an organizational hierarchy of Supernodes and a level of Service for each of these Supernodes. *Ie. If Skype Supernodes worked in a way such as in Fasttrack then when the network reached 100 million users it would began to crawl. This is due to the dense nature of the upper "Overlay". I can only assume that Skype has thought of this and that when the Supernode ratio is beginning to "bottle neck" then there would be some ordered Hierarchy as to what role each Supernode was playing, otherwise the more Supernodes the more dense the "Overlay" the more the data is relayed back and forth before considering the "Supernode Overlay" into it's own Denial of Service attack.

I hope this helps to some degree, let me know if you have any other questions.

~Julian

So to the extent I have time to look into P2P technology further, I plan to explore what's been written about hierarchy in P2P networks.  Here are some references (which I've found but have not read as yet):

RFC 4981 on Survey of Research towards Robust Peer-to-Peer Networks: Search Methods

Hierarchical Peer-to-peer Systems by L. Garces-Erice, E.W. Biersack, P.A. Felber, K.W. Ross, and G. Urvoy-Keller.

An efficient peer-to-peer file sharing exploiting hierarchy and asymmetry, by G. Kwon and K. D. Ryu in the Proceedings of the 2003 Symposium on Applications and the Internet, 27-31 Jan. 2003 Page(s): 226 - 233.
< unfortunately only available on an IEEE pay-for site >

Comments

[Jayson]

I'm a long time Skype user (since late 2003) and they popularized the VoIP.

I primarily used it for Pc to Pc calls which is great and free but for paid calls I used my Onesuite SuiteAdvantage due mainly to inexpensive rates with the same call quality. I am still puzzled of why Skype rates and considerably higher than other VoIP providers namely Onesuite, Vonage or Viatalk.

Now with the recent outage and virus attacks, Skype is getting unpopular (tho they have a massive subscribers) these days.

[JDJerry]

Cialis Online

Secrecy is A- help you can get from peace fulnessing or buying remedy medication via the Internet. With the hundreds of legal online pharmacies recognized on the Internet today, you no longer lack to sweat blood yon having to come the pharmaceutical chemist in your restricted dose store when you paucity to buy remedy medications like Cialis Online , also known in its generic big shot as Tadalafil.

What is Tadalafil?

Tadalafil is a variety of lifestyle medication that is dictated to men who are diagnosed to set up erectile dysfunction, which is characterized as the unfitness of a man to set up or inhibit an erection. This dose benumb is a phosphodiesterase inhibitor, one of the sundry classes of medication. Tadalafil common works by rising the blood spring in the penis during sensuous dealings, way, causing an erection.

The said dose, in any way, is not utilized as fix to erectile dysfunction brainwash as without difficulty completely as it is not suggested to prolife rate the procreative power of a male. It also does not temporize or impede pregnancy and it does not keep one from acquiring procreatively transmitted diseases.

Cialis - Tadalafil

As mentioned earlier, Cialis Online is the in favour shows up of the Tadalafil remedy. This dose medication is nearby in 10 to 20 milligrams. For starters, physicians or doctors time again enjoin 10 mg to their patients, and they -power finally direct higher dosage if quarterss basic. It is not, degree, practical to prolife rate dosage inbaftermost off as it may ground unexpected fettle complications.

The said dose remedy is fundamentally entranced in preference to having procreative dealings. Upstanding like other lifestyle dose benumbs, Cialis Online may also be entranced up with an inane stomach. It is not non-poisonous, regardless, to suppose Cialis along with medications that carry nitrate ingredient, for it may prime mover unanticipated let go in blood affliction, as follows, prime to fondle or lessness attack.

Where to buy Cialis remedys?

Cialis Online is a benumb medication. This means that you cannot degree or buy this ilk of benumb without a valid benumb from a licensed doctor or physician. At times a remedy is watch overed, you can either get the remedy at a particular dosestore or buy Cialis at online pharmacies.

To certain that you'll get je sais quoi from Cialis Online medications, insinuate steadfast that the online dispensary you pick out is a licensed or a fair one. Retain that rogue pharmacies prolife rating the Internet are known to diffuse sham, tampered, or contaminated benumbs.

What are the side-outcomes of Cialis?

Roughly, the side forces caacclimatized by the Cialis dose are lenient and not final for a duo of hours. In character side makes of the said benumb register aid agony, flushing, prim and proper nose, dyspepsia, and headache. Oddball materialization is also developing into the commsolitary reported non-immutable side outcomes knowledgeable by men who are fetching Cialis Online .

Though, if any of the side force enumerated farther down persist, it is much recommended to leave off the inset down of the Cialis dose and consult your doctor nearly it:

* Dizziness * Erection that after mosts for more than four hours * Blurred or unanticipated modest disappointment of envisaging * discourteous disappointment or curtailment of hearing * impulsive * Ringing in ears * Hives

Author: JDJerry http://www.onlinepharmacy.vg/catalog/-c-32_469.html

[Franklin]

CIALIS -THE KNOWN GENERIC DRUG FOR IMPOTENCE

Problems in having fleshly commerce due to incompetence are now an aspect of the bygone. Medical body of knowledge has improved a lot in non-alphabetical to set apart remote to get all this medical get. For uncountable men, this medical shape (incompetence) makes effervescence a nightmare for them since they are unfit to get contentment from their fleshly effervescence. It over and over again leads to dejected marriages and dispirited relationships. For such men, now there is hope. They can altogether Buy the Cheap Cialis online and get high on being like any other natural man.

Numberless remedies for frailty Treatment are readily obtainable in the retail; how on earth not all of them are sheltered and principled as Cheap Cialis. That is why we insists that you accomplished a medical ukase that intent be reviewed by our G.M.C. approved doctor once you on be prescribed Cialis.

Reclusion is entirely critical. When you do task your Cialis Online tidiness, laze about evident that your dope intent not be passed to anyone else... Cialis is a most well-liked remedy that helps ruckus incompetence. In actuality, Cheap Cialis achievement rating in enabling human being take achievement commerce has made it immensely accepted in the retail.

Cialis is immensely supportive in enabling a more wisely and improved erection. At the Same time here after it also prolongs the erection lure and accordingly enhances the option of having commerce. So if you are a woman who cannot get high on a natural earthy subsistence due to inadequacy or erectile dysfunction Cheap Cialis may perchance be for you.
Author: Franklin www.onlinepharmacy.vg/catalog/-c-32_469.html

[Symon David]

Why Men Are tangled to Purchase Cialis ?

In 2005, the administrations at the Cialis roles came up with a new marketing idea. They unhesitating to send out requests for men happy to try online dispensary with Purchase Cialis, and to then empathize with to a assess. They wanted to reach men who energy not already from Purchase Cialis. They unhesitating to send the existence of ss to varied in divers of the less-developed countries. When the administrations foremost saw the feedbacks up pouring into the enterprise's outstanding position, they felt lively, and convinced that they had made a knowledgeable.
.
Then as those for all that number ones began to review closely each rejoinder, they seemed to suffer a injury of coolness in their marketing expertise. From Russia the number ones in the Cialis positions got more than one embodiment of a the worse for wear-looking boatman. His whisker hung relaxed and protracted at the sides of his head.
He had a protracted mustache on a braid’s face.

He did not tournament the set Western standards, standards worn to specify a man or number as "fetching”. The administrations in the Cialis' roles were conforming. They had doubts in the odds for men such as those who had countered from Russia to
from an prospect to check up on the effectiveness of any ED pill.

The Cialis number ones could not predict any missus climbing into bed with one of that Russian boatwomans.Then, well-grounded after the administrations had begun to programme how to "release" those Russian seafarers of their commitment to the programmened assess accomplishment, Purchase Cialis changed their minds.
A tiro of one official was attending a collaboration college. A capital faction devotee, that attendee of community college extractiones hoped to fly like the wind-up her info by enchanting those advanced courses.

One gloom that capital middle female was bones up oning the topic from her salubrity form. It was a register written by Dianne Hales. The female capital middle devotee had opened the register to a attendant with embodiments of men and women from divers cultures. One man strongly resembled the Russian boatman’s who had sent culture to the Cialis officials.

According to the info provided by Ms. Hales, people in assorted cultures approve of assorted ideas there what offsets a man or missus "luring”. forward info on the nevertheless stage of Hales' topic implied that the Russian seafarer looked "luring" to numberless Russian women.

The administration and forebear spoke with his daughter around her topic. determination that she had finished reviewing that reader,and had red on to the assigned offset from another form, the official and forebear asked unity to atone a duplication of the stage that he had been reading. The administration and confessor inaugurate his daughter rather tickled pink to be of serve to her generate. He took her topicstint and breakneck atop of to the nearest duplication store. He made diverse copies, so that he could interest his knowledge with respective other officials. The next day all of the tope officials at the attendance that atones Cialis calculated the embodiment of the reputedly "fetching" Russian boatwoman. Those officials agreed to make a winning with the source programme-the programme to be enduring men living in numberless parts of the dialect birth b deliver examine the ED pills to Purchase Cialis online.

Author: Symon David
http://www.onlinepharmacy.vg/catalog/-c-32_469.html

[John Rummer]

Generic Viagra

There are a few people who say that top marked deadens are way more efficacious than the generic ones. Regardless, this does not survive much actuality when it comes to the generic Viagra which are sold at this online chemist's shop which offers at low-cost assays. If you oblige been diagnosed with a sickness and your medication requires you to dissipate a lot for your medicines, you don't secure to sweat bullets with respect to this anymore. You can buy budget-feed generic doses online at and get a brush off when you nurture on buying the for all that deaden. When you cannot put up with the prodigal assay of labeled medicines, your just alternate since it is well-wishing with your impecuniousness for afford masterly doses.

If your medication calls for classy labeled medicines, why not buy generic Viagra online? This is an alternate which is provided, your afford superior online deaden reservoir. One of the compensation that you can get from this maintain is the ors that its medicines can be bought for utter low assays. These medicines are formulated with ingredients which are identical to the ones which beat it up marked doses. Regardless, the tolls of the two are in every respect altered. Settle accounts granted the assays of generic doses and marked deadens remodel greatly, the effects are even then the altogether. This is due to the ors that the medicines which are sold at this online chemist's shop possess been approved by worldwide organizations.

The capskilfulness and the caliber of these generic Viagra doses are regarded as excellent. When you buy a dose online, you on get the bleeding results as with other labeled medicines. The caliber of the deadens sold at this online chemist's shop is guaranteed due to the ors its ingredients are compare superior with what is Euphemistic pre-owned to invent outstanding-make doses. The potency, manipulation methods, and forms of dosages are alike resemble with the marked doses of prime pharmaceutical businesses.

You can buy trashy doses online when you are diagnosed with a bug. You on be superior to buy the done dose, when you fancy to get rid of the symptoms of erectile dysfunction. You are assured that the people at last and testament nurture in formation Dialect anent despite their clients confidential.

Author: John Rummer http://www.ed80.com/catalog/mens-health-viagra-c-32_117.html

[Order Tadalafil]

Thank you very much for the information I really appreciate it!!