First let me summarize my problem. When SIP emerged in 1996, it's support for direct connections from one user to another was extremely compelling. This was the VoIP protocol which would lead to a complete revolution in communications. Yes, you might refer to a directory service, but you wouldn't need an operator to make a phone call. You could do it yourself, directly. Unfortunately, that revolution never happened.
So far, no revolution
The biggest change in telecommunications in the past 12 years has been the global deployment of three billion mobile phones, all based on conventional circuit-switching and Intelligent Network technology — nothing to do with SIP. And arguably, the most interesting telephony service enhancement, after mobility, came from Skype with its seamless integration of presence, instant messaging, wideband audio and video. But Skype is based on proprietary protocols, not SIP. Finally, VoIP technology has helped drive down the cost of international calling, but using MGCP, H.248 &/or H.323 protocols much more than SIP, at least so far.
SIP has been adopted by PBX manufacturers in recent years, but this doesn’t seem to have changed business practices at all. The IT department still buys the PBX and the telephone sets from a single vendor and then contracts with a service provider to handle calls outside the enterprise.
And then there's IMS
SIP has been adopted for use in the IP Multimedia Subsystem (IMS), but this completely warps the original SIP vision. IMS is a centralized system — a next generation network for mobile and fixed operators. It's the complete opposite of the original vision for SIP.
Why have things gone so far astray?
SIP assumed an end-to-end Internet
SIP assumes it's possible to make end-to-end connections over the Internet and therefore a SIP session can know about and use globally valid IP addresses. That was a naive assumption, even in 1996-1999 when SIP was being defined. The real Internet contains firewalls, network address translators (NATs) and other "middle boxes." They are not going away, it's only getting worse over time. Today, applications must be aware of and able to work around middle boxes and other network problems.
Many middlebox issues can be overcome with the help of client software and central servers implementing Interactive Connectivity Establishment (ICE), a recently completed IETF proposed standard that in turn relies on STUN, TURN and/or RSIP. A continuing obstacle for direct user-to-user connections is the need for central servers for STUN, etc..
So it there no chance for the original SIP vision of direct user-to-user communication?
P2PSIP — a reason for optimism
Actually, there is some reason for optimism. The advent and widespread adoption of Skype showed what was possible and suggested how one might distribute central services among peers, potentially avoiding the need for an explicit service provider. The past few years have seen rising interest in peer-to-peer SIP which has resulted in an IETF working group under the name p2psip. Their goal is "to leverage the distributed nature of P2P to allow for distributed resource discovery in a SIP network, eliminating (or at least reducing) the need for centralized servers."
Assuming this is completed (during 2008 & 2009), we'll have the elements with which one could make a SIP-based open peer-to-peer communications system. It will be interesting to see actual software implementing the ideas of the p2psip group. We may yet see a revolution!