The Internet started as a distributed network – ARPANET. Its protocol, TCP/IP, was designed so that hosts anyone could communicate without relying on a central authority, and if any one host went down, it didn’t affect the other others’ ability to work. Gradually, as ARPANET evolved into the Internet, that distributed architecture became more important. Today, the Internet is pretty much everywhere, from nanny cams, to laptops, pda’s, servers, data centers, and telephones. And that redundancy and distribution is critical for the whole thing to work.

Great… so I’ve just wasted a paragraph telling you what you already know, right? The interesting part is that business doesn’t like distributed services, and has been trying since the beginning to stop them. For example, the World Wide Web is distributed. Anyone can set up a web server, and then anyone else can view pages on it. I can set up a web server on my PC and someone in China can take a look around. No reliance on Google, or Microsoft, or anyone else. But then, how do businesses such as Google and Microsoft make you rely on them, so they can charge you for their services, or force you to view advertisements, so they make money?

We have this constant struggle of the technology pushing towards decentralization, and the business interests pushing towards centralization. Let’s take Amazon S3 as an example. S3 is a very cool distributed computing technology. The basic idea is that one can write software specifically against S3’s API and then upload it to S3, and the service will dynamically provide more or less resources depending upon the site’s specific needs at any given time. If the site is hit by digg or slashdot – that’s fine, S3 will allocate more processing power. When the surge ends, the power is scaled back, and the payer of the S3 bill saves money.

So far so good, right? But now there is a situation where a vast number of sites have a single point of failure. They also have no portability to other providers. What if S3 goes down? All the sites relying on it go down. What if S3 raises their prices? The sites can’t change providers; they have no choice but to pay.

S3 is relatively new, so the technologists haven’t yet come up with a complete alternative, but some are in the works. But, consider Twitter. Twitter is a microblogging service that allows one to write 140 characters of one’s thoughts. It’s tremendously popular, having over a million users worldwide. But what if Twitter became unreliable? What if they stopped introducing new features, or broke existing capabilities? What if their terms of service weren’t to your liking? The answer to all these questions used to be “deal with it” or switch to another site just like Twitter, that inevitably evolved the same problems. Twitter isn’t distributed – you can’t just make your own Twitter and still communicate via your new site with your friends on Twitter (like how an GMail user can email a Molecular address), nor can you contribute the Twitter software. Your complaints fall on deaf ears.

It took a while, but the technologists responded with Identi.ca. Identi.ca lets anyone download the software the site runs (called laconi.ca), make changes, and run their own server. If identi.ca becomes unstable, the owner starts behaving badly, or anything else disagreeable happens – the users can simply leave. Their data comes with them, and they can still communicate with others on the OpenMicoBlogging network. Imagine that… if you don’t like the service, you can leave.

There are many examples of services changing from locked down to distributed, too. Instant messaging is one: there used to be many many protocols and clients people had to run, such as AIM, Y!, MSN, etc. Then XMPP (aka Jabber) came along. With XMPP, one provider going down only affects those users and not the whole network. For example, when AIM has trouble, every single user of AIM has trouble. When GTalk goes down, I only lose communication with those people on GTalk, I can still talk to my friends on other XMPP servers (for example, when the aol.com email server goes down, it doesn’t affect my ability to send email to gmail.com users from my Molecular address). The damage is limited, and people have choice. Those users of GTalk, who are now seriously annoyed with the service for going down, leave to another XMPP service, and they can still talk to all their friends, just the same.

Microsoft Passport failed for the same reason. People didn’t want to rely on Microsoft alone for their ability to log in to disparate services. Instead, OpenID came along, which allows users to pick, and even change later, who their identity provider is. And now bigger players are supporting OpenID.

These distributed services provide a huge incentive for service providers to keep providing quality services, that are stable, work well, and continue improving. With the constant threat of users simply leaving, these services must always be on the look out for improvements, and they can’t simply add tons of advertisements, or paid registration, when looking for money.

In the short term, the Twitters and AIMs of the world will continue to have huge user bases. But over time, they can’t keep up with the freedom and higher quality of the distributed alternatives. Look around – is the centralized Prodigy still around? How’s Delphi doing? For that matter, isn’t the walled garden of AOL in trouble? Yet the World Wide Web and the email system keep going.