Web Architecture

A Web-Oriented Architecture (WOA) Un-Manifesto

Dion Hinchcliffe — Mon, 14 Dec 2009 11:31:00 GMT

With the service-oriented architecture (SOA) community recently issuing its SOA Manifesto (based closely it seems on the largely excellent example of the Agile Manifesto), I have been thinking recently about whether we need such a document for Web-oriented architecture, also known as WOA. The manifesto format can be a useful tool for creating critical mass around and focus on an important new idea or approach. I don't believe, however, that we need one for everything that comes out; only when important new advancements are still in need of clear explanation. So while the reception was decidedly mixed for the SOA Manifesto, by-and-large I do think such efforts can create useful statements of principles that get people thinking about the fundamentals and underlying truths of their work and help in creating a shared understanding.

The challenge with broad declarative statements like a manifesto is that technical communities are by definition consist of technical people. Such people are typically deeply experienced with formal intellectual appraisal and are famously independent thinkers who in my experience generally respect open debate and discussion. But they will almost automatically resist when ideas appear to be predigested for them and then presented on a platter. This is particularly the case if a hidden agenda or conflict of interest is perceived. Thus, the manifesto approach is a useful one most often when it's created through a collaborative effort and with concepts that have previous been subject to widespread discussion. It is less effective when it's used to distribute propaganda or tied in with motives that are unclear. In other words, it shouldn't be done lightly.

So I debated at length over the last few months whether WOA actually needs a manifesto. In the end, for reasons that I explain below, I finally concluded that it does, just not in the traditional way. I'm calling the output of this new approach an "unmanifesto". You'll find the 17 principles of the WOA Unmanifesto listed at the bottom of this post, each of which will be explored in more detail in upcoming posts.

Skip straight to the principles of the WOA Unmanifesto

Backstory on the evolution of WOA

Personally, I've been fortunate to spill quite a bit of fairly popular digital ink on the subject of WOA over the last few years. I did this because I felt information availability on the topic was fairly poor despite the central importance of the concept to most efforts involving software architecture these days, either on the Internet or in the enterprise.

Of course, Gartner's Nick Gall is the man most directly responsible (to paraphrase a line from a great sci-fi film) for creating the initial concept of WOA. I was fortunate to meet him present personally at an enterprise architecture summit many years ago now and see him present the concepts. I was captivated at the time and I remain captivated by WOA today. Since then, I've been fortunate to help carry the banner and put my own spin on what I think WOA is and where it's going. I've not had as much time to sit down and compare notes as much as I'd like with Nick and other great REST/WOA thinkers like Roy Fielding, Sam Ruby, Tim Bray, and literally hundreds of others that I must leave out for brevity. This is an attempt to start that discussion in earnest since WOA is clearly a growing influence on a great many aspects of software architecture today.

But ensuring we have a broad understanding about what WOA is part of the problem. My premise from the beginning has been that we can use the example of the Web itself to extract enormously important lessons about Web architecture if we are willing to learn what it has to teach us. WOA at its core is based firmly on REST principles, but that's only the beginning since the Web continues to evolve rapidly and we've seen important patterns and additions emerging such as mashups as well as OpenID and OAuth, to name but three major developments. But as I've covered in the past, while REST is the canonical basis for building Web-based software when it comes to moving data around the network, we still need a bigger conceptual umbrella to capture the higher order architectural advancements in Web software (i.e. those building on REST) . This broader set of essential patterns and concepts is one that has co-evolved with REST and the Web (including distribution, composition, security approaches, data models, data portability, and so on) over the years. WOA is that umbrella.

WOA is more than REST or ROA

Though the underpinning is REST, WOA is therefore a superset of resource-oriented architecture (ROA) and contains all the moving parts we'd find in an architecture: 1) the basic elements of resource-based architecture, 2) the glue the binds them (HTTP transfer is the core one, but there are refinements now, e.g. ATOM and JSON feeds between browser widgets for example), 3) consumption models (how browsers and other clients fit in, mashups, recognizing OAuth as a first-class citizen in Web architecture, etc.), and the patterns that connect them all together.

First, though, to get perspective on all this and to help us decide what's really in WOA and what's not -- something that I've tried to do for several years now -- we have to define some generally agreed-upon principles to give us a set of objective discriminators. Let's not forget that the Web is one of the most effective systems for connecting people, information, and software yet created. It has profound and fundamental properties that are enormously powerful when understood and used effectively. These can be fully unleashed when architectural principles are followed that are aligned with the Web's fundamental nature. Fortunately for us, many of these principles have been discovered or developed over the last 16 years or so of the Web. Tragically, they are as not well known as they should be. Part of the reason for this is that the Web itself is an emergent, bottom-up phenomenon and it's not always easy to see what's happening on a wide scale. This bottom-up state of affairs has held back the development of the Web to the extent that most people don't realize that WOA, by definition, is the dominant software architectural model in the world today. It's what's being used today and is what works most effectively, most naturally, and most easily when it comes to the overall design of Web-based systems.

But "effective", "natural", and "easily" are subjective terms. How can we get more specific about defining WOA? Can we reach a genuinely accepted common understanding that would allow us to decide if a software system that we encountered did or did not use WOA concepts? Most importantly, how do we look at new innovations in Web architecture and decide if they are part of WOA or not? There needs to be a dividing line. Something like a manifesto might help.

What's an "un-manifesto"?

To create this dividing line, I am issuing something I'm calling here an "unmanifesto." A manifesto developed by a single person is by nature of limited value, since it should instead reflect some wider conversation that represents a broader, community-validated understanding. Like an unconference, where the contents of the agenda are arrived at by a collaborative process and the attendees are also presenters, an unmanifesto is a distributed one in which there is more than one voice and the statements are made in public, online via social media. I call it an "un-manifesto" because it's not a manifesto, it's just the start of an important conversation by making some specific statements. An "unmanifesto" is an open, online discussion where it's assumed that there are multiple points of view. Anyone can join in and no one is excluded. Sound confusing? It is a bit messier than a traditional manifesto but it's also far richer and more open. If you care about modern Web architecture, I hope you will join me in this effort.

Thus, the principles below represent my WOA unmanifesto. There may be other WOA unmanifestos. This one will evolve over time as we learn more. Of course, anyone may issue their own WOA unmanifesto and anyone may comment and critique the ones that are created. The best ideas will be debated, survive, and be replicated. The ideas that aren't good just won't last. Eventually, after a distributed and open discussion, clarity will be achieved. Of course, this will only work if a community exists that is interested in the topic and is willing to actually listen to each other. I hope there is and that we will.

The Web-Oriented Architecture (WOA) Unmanifesto

The principles are broken down into four major groups: Resource-Orientation, Performance and Consumption, Interaction and Composition, and Openness and Security.

I. WOA Is Based on a Resource-Oriented Hypermedia Data Architecture

Principle #1: The fundamental artifact and unit of value in WOA is data stored as uniquely identified Web resources on the network with a uniform interface based on REST.

Principle #2: Web resources are maintained in a hyperlinked fabric using hypermedia (content with non-linearly navigable embedded links). In general, URIs should favor granularity and depth in linkage instead of being monolithic.

Principle #3: Resources and their network addresses (URIs) are self-descriptive so that WOA clients can determine how to consume them. In general, a URI should indicate what data format is being used and indicate nested elements with URL segmentation.

Principle #4: All application data is therefore accessible as resources in the manner presented here. Resources are visible on the network to authorized WOA clients, where sharing to all is the default should be the default whenever possible. Note: The major implication of this, particularly for legacy environments (enterprises) is that all existing databases and other datasets should be mapped to the resource model described here via REST. Even servers, storage, and network infrastructure are considered resources as they increasingly are in cloud computing APIs today (see OCCI).

Principle #5: Thus, resource-orientation is the best way known to 1) create a loosely-coupled, participative and shared global architecture that 2) can scale almost infinitely, 3) be used by virtually any platform or client, 4) directly facilitates information discovery (by analyzing link structure), 5) be easily recombined, and last (not least) 6) builds value non-linearly in the form of network effects.

II. Performance and Consumption: First-Class Citizens of WOA

Principle #6: Web resources must be stateless, layered, and cacheable, as per REST.

Principle #7: Network consumption of resources (transfers) is inversely proportional to how complex a Web resource is; i.e. the simpler the structure and data of a resource is, the more likely it will be used, recombined, and perform well. See the Services Continuum discussion for more background.

Principle #8: Resources should organized to optimize their discovery and reuse. This principle has many more implications that it sounds like at first. Not only should resources be SEO-friendly but it must be recognized that they are also APIs that support applications elsewhere on the network. Optimization for the latter can mean licensing agreements, SLAs, and customer support and all the trappings and requirements of open APIs.

III. The Structure of the Web for Application Architecture and Value Creation: Interaction and Composition

Principle #9: A resource is recognized as having a higher realized value (network effects) the more that other parts of the network link to it.

Principle #10: Copying of resources is discouraged: The network effect of Web resources is increased when they are linked to from other Web resources instead of replicated. Caching is encouraged however, as long as it's at the same URI.

Principle #11: WOA clients provide value by connecting to the network to view, interact with, operate on, and recombine resources from multiple sources using REST. Note: The browser is the canonical Web client. There are many other types of clients.

Principle #12: Observable value is increased when resources from simultaneous multiple domains are accessed by WOA clients (mashups) which are themselves resources (self-contained applications).

Principle #13: Self-distributing applications create additional network effects. Native WOA clients can be distributed as resources (code-on-demand) that are self-contained applications (such as AJAX, Flash applications, widgets, etc.) to be embedded in elements of other WOA clients. This increases in turn the value that the resources the WOA clients use.

Principle #14: WOA applications inherently encourage link propagation, both in the resource fabric and out-of-band, through syndication, widgets, badges, mashable components, open APIs, and any other method that invokes Jakob's Law.

IV. Openness is Balanced with Security

Principle #15: Resources are in formats that are based on open, published descriptions and standards. Resources must be readily consumable from the canonical WOA client.

Principle #16: Resources can be secured with HTTPS, WebDAV, OpenID, digital signatures, etc. Locally defined user ID and passwords are permitted but discouraged as are proprietary (i.e. not open standards-based) security approaches.

Principle #17: Resources that are secured must be exposed to WOA clients by a safe 3rd party authorization method. OAuth is preferred.

These 17 principles represent a significant update to my original definition, which remains widely cited. WOA has evolved and our description of it must evolve with it. Please note that I have pushed some details down into the description of the principles, which will be forthcoming soon.

I hope you see how there is more to WOA than just REST and that there are clearly higher order principles around composition and distribution architectures that need more definition. On the composition side, for example, incorporating Michael Ogrinz's mashup patterns may be a good way to access best practices and eliminate duplication. I'll be exploring that as well as other sources of high value inputs as this unmanifesto evolves. Keeping the principles clean, well-organized, and at the same level of abstraction will important and I'll do my best to achieve that. Also, I'll be clear that I still firmly believe that SOA itself is largely headed in an WOA direction, it'll just take a while given the momentum and vendor history in that space. So the WOA Unmanifesto is just as relevant and applicable to the SOA world as the Web world.

Finally, there are many essential details and implications for each of these principles, which were not included here lightly; each and every one is a key aspect of Web-oriented architecture. I also realize that for these principles to be broadly understood and themselves consumable by most Web architecture, each and everyone one of these principles needs a detailed breakdown with examples. This is one of the big problems with WOA being an emergent architectural style and not a standard in itself.

Over the coming months, as often as I am able, I will be breaking down each of these components as we begin discussing them here. Input and feedback is very welcome and encouraged both in comments below and posts (these are preferred) and by e-mail (mine is dion@hinchcliffeandco.com) only if necessary. Also, I'm reserving the right to tweak this for a few more days to make sure the wording on the principles reflects my intent. I'll put a change log below to make sure nothing is lost.

Please join in the WOA Unmanifesto discussion below. I will respond as often as I can.

Further Reading:

What Is WOA? It's The Future of Service-Oriented Architecture (SOA)

Where Is The Future of SOA Headed? Where The Web Goes...

Unboxing Web-Oriented Architecture: The 6 Aspects Of An Emergent Architectural Style

Web 2.0 success stories driving WOA and informing SOA

The SOA world begins considering Web-Oriented Architecture (WOA) in earnest

Can We REST For A Minute? 6 Lessons From The CRUD vs. Hypermedia Debates

Dion Hinchcliffe — Thu, 06 Aug 2009 12:31:00 GMT

Recently InfoQ did a good summary of the debates around the apparent (to some) limitations of REST when it comes to creating good Web services. At issue is that REST APIs seem to expose "CRUDy" services that fly in the face of years of good services design, particularly when they are just read/write interfaces instead of the richer, full REST architecture (more on what this is later.) The discussion was spurred by Arnon Rotem-Gal-Oz's assertion recently that CRUD is bad for REST, which in my opinion is close but not quite right.

In my view there appears to be two primary ways that REST are used today. One is via the aforementioned CRUD approach (where CRUD = Create, Read, Update, Delete) and HTTP's verbs (which in the same order as CRUD are POST, GET, PUT, DELETE) are used to manipulate a data resource on a Web server. This is the most basic, most common way REST is used by application designers (sometime even using these to replicate traditional RPC) and is the foundational request/response model in REST services. Unfortunately, it's also probably the least interesting or powerful way to apply the full REST architectural style.

For one thing, using only CRUD to manipulate data treats resources as loose, unintelligent bags of bits, seems too fine grained, and generally feels like client/server in a cheap disguise. Sure, it works extremely well with the protocol that runs the Web, HTTP, but no architect would consider this story so far good enough reason by itself to bend an entire application design around it. Things get much more interesting, however, when URIs start getting added to the mix. This happens in one of two ways and, depending on how URIs are applied, it's the key to the whole REST hypertext approach (aka HATEOAS, or "hypermedia as the engine of application state") which has worked so well on the Web for content. And now, with REST, we're doing it with data.

Data as Hypertext: Deeply Linked URIs and URIs for State Transitions

First of all, like the Web itself, REST works better when the data itself links to URIs of other resources that contain a more complete representation of a given piece of data (a line item summary might link to the full line item resource, for example). Deep linking, and not maintaining local copies of a representation is essential to prevent a variety of issues including data duplication and out-of-date copies.

Second, the very essence of REST is to make the states of an application explicit and addressable by URIs. The current state of the application state machine is represented by the URI that the client just operated on and the state representation that was returned. The client can change state simply by operating on the URI of the state it wants to move into, making that the new current state. A given state's full representation includes the links (arcs in the URI graph) to the other valid states that that can be reached from the present state. So in the visual below, we show a simple but complete interaction model that includes a single valid state transition. In short, as even Roy Fielding himself recently emphasized last year, REST APIs must be hypertext driven.

Click To Enlarge

The scenario above shows a complete and I hope canonical (albeit simple) interaction model that exhibits the full RESTful properties. The individual numbered steps of this scenario are described below:

Step 1: To use a well-designed REST API, only a single well-known URI needs to be published by the creator and referred to by the client. From this top level resource, all the resources and state transitions for the entire API and all its services can be found. A REST API is self-descriptive and therefore self-contained, with only minimal "out-of-band" documentation required. A simple HTTP GET at this URI will retrieve the API entry point.

Step 2: The API is examined and a service known as FooService is found, along with the URI for the entry point to the FooService API.

Step 3: The resource list for FooService is retrieved via GET. The URI came from the top-level API description examined in the previous step. In this example, a list of fooitems is retrieved, including possible state transitions.

Step 4: There is only one state transition, "reverse". This operation is chosen to be used in this scenario.

Step 5: A PUT (which means the resource will be modified) is issued to the URI described by the original resource. A back-end component (business logic) then reverses the name element of the resource (changing the state of the resource). A copy of the modified resource is also returned by the operation, making the operation efficient as well.

Step 6: Although a copy of the resource was returned from the state transition, the change is durable but we do another GET, this time just on the resource to see what the current state is.

Step 7: The state of the resource is examined and found to be changed by the state transition operation.

Lessons learned from the CRUD vs. Hypermedia debate

So, I think one of the biggest problems with REST is that there is no canonical description for how to use it. This is also one of its core strengths. There are many ways to apply REST to the problem at hand, and if all you need is CRUD, you are still doing basic REST and meeting your needs. No single description could do justice to the power and flexibility inherent in the model and I agree strongly with Sam Ruby that "REST isn’t an all or nothing proposition. One can get significant value from partial adoption."

But this discussion does highlight some very specific lessons learned that many REST practitioners don't often follow but are important to consider carefully in order to build services that are resilient, adaptable, and maintainable over the long term. Many of the benefits of REST aren't felt until later when you want to fix or change things. That you can do so easily is a testament to the potency of architecture of the Web, if only we can understand it clearly enough.

Here are my takeaways on what most REST adopters can and should do to get the most from their use of this increasingly popular architectural style:

Describe your REST API via a single, well-known, top-level resource at a permanent, public URI. This resource contains descriptions and URIs of all the services (by type) that are maintained by the REST API. This provides maximum flexibility for changes down the road since well-written client code will process the changes as normal operation and not require specific alteration for many kinds of modifications to the API.
Use business logic behind the server CRUD, return HTTP errors as appropriate. This one isn't as well known as it should be but REST services do not have to be dumb repositories of resources. They can do as much validation or processing as required for all POST, PUT operations. Don't accept data blindly from clients and return HTTP errors that make sense and describe the problem. Does this violate the self-describing aspect of REST interfaces? Not necessarily, the server imposes the valid state transition on the client and this is just another means of doing so.
Never hard-wire URIs into the client, always assume the resource will provide the current URI to related data as well as valid state transitions. Good REST interfaces will be easy to maintain for the server, but clients can take shortcuts, either to improve performance or to simply coding. Don't do this, just dereference URIs every time and you'll pick up and use modifications to the interface largely automatically and with no code changes.
CRUD isn't the only way to change a REST resource, invoking a state transition's URI can modify data implicitly. This is where REST can become very elegant and provide a self-describing interface and state model for changing data. Designing state models aren't what Web developers are particularly familiar with but it's a potent programming model that encourages good discipline and works for most types of applications. You can use CRUD where state transitions just don't make sense.
Versioning is best done outside of URIs. Version all day long but don't hardcode it into the URIs themselves. This can easily be done with content negotiation. You can also read Paul Sadauskas great explanation of how to do this this well.
The art of describing state transitions is still very much an art, study the latest conventions. There seems to be little consensus on how state transition URIs should be documented. Does one use the href tag or the uri tag? How does the resource convey exactly what the transition does in terms of altering state. This and other questions are all relatively unanswered still, though they are fairly obvious to those that examine a given REST resource. There is more consensus, for example, on how to pass parameters into state transitions or operations such as searches and queries. This is where URL Templates have made some headway and it's important to study them.

I've frequently described REST as a foundational element of Web-Oriented Architecture for both enterprise and Web applications and open data in particular. It's been thrilling to see REST become so popular in recent years, even if we have a long way to go yet to reach a wide understanding of what it is and how to use it. I hope these ideas are helpful and I'd love to hear your additions below.

Unboxing Web-Oriented Architecture: The 6 Aspects Of An Emergent Architectural Style

Dion Hinchcliffe — Sat, 06 Jun 2009 15:16:00 GMT

So you've been reading the many recent articles on the Web about REST, how it's the future of Internet architecture, and you've decided to begin applying some of it in your work. But when you start digging into the subject, REST by itself seems to be about moving data resources from one place to the other over the Internet, not building complete applications. It's unclear (and certainly poorly documented) to you how to situate it properly in context; a fully realized and well-designed piece of software.

You start wondering how REST is used specifically (and strategically) in an application or overall software architecture. In other words, what are the moving parts and rules for applying them. You also wonder what else you should aware of that can be used in conjunction with or to complement REST and make it better. You might even want to know when to break the rules and use other related approaches. This is where something known as WOA becomes useful.

WOA as a complete REST architecture

This is where a deeper discussion on Internet applications and specifically, Web-Oriented Architecture (WOA) comes into the picture. WOA creates a more sophisticated and up-to-date vision for modern Web applications that aligns gracefully with the grain of the Internet. It also is generally (but of course, as with any approach, not always) true that WOA applications are easier to build, connect to other systems, and maintain for the Internet (and yes, Internet-type networks, like your enterprise intranet.)

Why is WOA generally so much better than traditional service-based architectures? Because WOA is an integrated, emergent architecture that is born out of countless lessons learned about what works and what doesn't when designing software for the Web. It's not created by a big software company, it's not a commercial piece of software, it's not a giant set of standards from a standards body.

Instead, WOA is just what developers are doing lately and more and more of it is showing up in our favorite Web application frameworks like Ruby on Rails, Django, and the nascent cloud computing platforms. In other words, it's a set of best practices for designing Web applications. And it's not a grab bag of approaches and technologies, it's actually a fairly elegant model that works: It's secure, it scales to the Web, it performs, it's all those other things that we like when we are idealized lazy programmers. WOA is also generally simple to work with, easy to implement, and usually is intuitive once you understand the Web. Note: One exception to this is REST being "the engine of hypermedia state", that certainly takes some genuine noodling over to understand. I'll see if I can tackle that in an explainable way in the near future, since it's one of the best and most important parts of REST. In the meantime, a walkthrough of Sun's RESTful Cloud API perfectly illustrates this concept.

So, in short, WOA is much more than REST and REST is the foundational architectural style for WOA.

You can see all the key elements in the diagram in the upper right of this post. It's a good start at understand what's essential about WOA (which at its core is about RESTful approaches to software) and what else you can do with it. Chances are good you'll end up using things like JSON and ATOM. Perhaps to a lesser extend things like OpenID or OAuth, though you should definitely put those on your to do list.

A more exhaustive list of things that are WOA is below. Many of them can be used in non-WOA ways (in other words, breaking RESTful principles). Don't use them that way. Instead, think about resources, links, and hypermedia and how to compose, distribute, and consume them.

The Aspects of Web-Oriented Architecture

Transfer Methods. This is at the core of REST and ROA and is the foundation of Web-Oriented Architecture. You can read Roy Fielding's chapter on REST (he conceived of the approach originally) or you can just follow the simple guidelines here. At other times, protocols like BitTorrent can be used if the requirements warrant it, but these are exceptional scenarios that I will cover at some point in the future.
Data Representation. Just about anything that HTTP can transmit can be your state representation which you can then compose, distribute, etc. XML is standard and JSON is getting more and more popular but it can even be an image or other media, though in general, the simpler the representation, the more consumable it is.
Data Portability. Getting your data into and out of WOA-based systems requires some support for a few standards. In particular, being a good Web citizen requires paying attention to these, even if most users or customers don't ask for it upfront. Eventually, they will want it, need it. Fortunately, these are also pretty straightforward.
Security. Internet security is a major and ongoing topic and securing you WOA applications requires more than just SSL, which is really the only option with HTTP that is widely recognized and universally used. SSL has also never been compromised. But user identity especially is evolving very quickly on the Web right now and open security/identity approaches like OpenID and OAuth need to be in your architectural plans these days. Both are also very WOA friendly.
Composition. Building applications out of REST services is a whole set of techniques in and of itself. Hypermedia is of course the core model for composition and your Web pages or other code will operate directly on resources with RESTful design principles. Mashups and and Web widgets and gadgets are important too. OpenSocial and other emerging social networking application standards (which work best as, you guessed, WOA apps) are also becoming important as well but have some potential for lock-in and their eventual success is still unclear.
Distribution. Getting your services out there and consumed is still an art form, but the technologies are straightforward. HTTP is of course the canonical way to access REST resources but going the full API route is the best way to get them consumed. Widgets are thus a key distribution strategy (on the push side) in addition to being a composition strategy (on the pull side).

If you on the enterprise-side of the story and wondering how this will affect you, please read my recent examination of WOA and SOA.

Update, June 8th, 2009: Dave West over on InfoQ did a good summary of this post today in REST is a style -- WOA is the architecture.

Modern Web Identity: Why Your Web Applications Should Be Offering OpenID, OAuth, And Probably Facebook Connect

Dion Hinchcliffe — Wed, 15 Apr 2009 06:02:00 GMT

Are you creating a new Web site and developing a user registration system that requires new visitors to sign-up and create a user ID and password? Stop now and read this. There are now more effective approaches for dealing with Web accounts which are more powerful and are better for you and your users.

The concept of Web identity has recently undergone significant evolution that all Web developers and architects, both consumer and enterprise, should be readily familiar with today. These new identity options, specifically OpenID, OAuth, and Facebook Connect, when individually used can:

Make it easier, faster, and safer for users to establish their identity on your site.
Enable businesses to have single sign-on to your applications.
Turn your site into a platform for 3rd party applications which can access user data safely and securely.
Can integrate your site or application into the social experience of the user and their connections.

This post explores these three new ways to provide user identity that will make your Web applications work substantially better and make your users more secure and satisfied. There are also some important caveats and issues to be aware of and which we'll also discuss.

Backstory

One of the parts of the classic Web that's unfortunately still with us is the trusty old user ID and password combination, one per Web site. That's because the Internet itself has no intrinsic identity system built-in; anonymous access is the default and anything more sophisticated has to be cobbled together one-off for each site. This means most Web sites today implement user identity uniquely with different sign-up processes, user ID requirements (sometimes you get to pick, sometimes you have to use your e-mail address, and very often your favorite ID is already chosen, etc.) Then there are varying password length requirements if you are lucky, variable recovery processes if the password is lost, and so on. This is the challenge of an open, decentralized system (the Web) that has very little top-down design: How to provide users control and consistency in an online world where there are very few widely accepted user-centric open standards.

Interestingly, this is the one of the rare instances where enterprises got it right long before the Web and is now a relatively well-solved problem -- at least inside the firewall -- with something known as single sign-on. Often referred to as SSO and embodied by technologies such as Kerberos, smart cards like RSA's SecurID, and Windows Integrated Authentication, single-sign on ensures that users have just one single, consistent identity and sign-in credentials. Single sign-on works then seamlessly to enable them access to the various applications that they use on the network. There is only one user ID and password to remember and it can be reset, administered, or even shutdown centrally whenever required.

However, these enterprise approaches -- as good as they have become -- are generally unsuitable for the Web identity for several reasons; they are either involve proprietary approaches (which is anathema to gaining the support from the Web community which thrives on open standards), require special hardware, or are too hard to implement consistently across the hundreds of different languages, frameworks, and platforms that make up Internet the today. In fact, anything that isn't aligned closely to the way the Web works, recognizes the browser as the first-order Web client, and offers a lightweight, open approach that's easy to implement from most toolkits just won't succeed.

Note I say "easy to implement" is a requirement for Web identity systems and I should note that making user identity truly secure is one of the harder problems in software. This can mean that OpenID and OAuth require a fair amount of work to implement successfully. The good news: All three new identity approaches presented here are now fairly easy to use right out of the box since much of the hard work has been done to create libraries for the most common programming languages and environments. I will observe that it's still up to you, the designer of the Web application, to ensure these libraries are doing the right thing by your users. This is yet another reason why Web identity is still so often reimplemented over and over again; lack of trust of 3rd party code and not-invented here are still two powerful forces in software development. However, the value proposition has grown to the extent that these do-it-all-yourself positions are growing increasingly untenable.

That's not to say considerable care shouldn't be used. In general, the more important the user data your application handles, the more auditing and regular code reviews you should conduct of your security and Web identity libraries.

The identity options: OpenID, OAuth, and Facebook Connect

Each of the identity options presented here does something slightly different in terms of providing users with either their choice of login and Web identity or safe access to their data from elsewhere on the Web. As such, you will likely be faced not with choosing one of these three identity options but actually all three of them to give your users the options they what and will increasingly expect. I've highlighted in the past the number of core competencies that modern Web developers and architects have to master these days, and it's a tall order. These options add to that burden but as you'll see, provide considerable value in return.

OpenID

The best and most authoritative explanation of OpenID can be found here at OpenID Explained. It's quite simple: Let your users use the identity provider of their choice to login to your application instead of filling out an online form and capturing the information in a private, local account that users are not likely to trust nor long remember. Chances are nearly 100% that a user already has a valid OpenID from the many popular services that already allow their IDs to be used this way today. OpenID use is also growing: Recent reports show that sites offering OpenID logins are currently reporting that 10-15% of users will login this way and the average is climbing. Venture Beat also had a decidedly bullish outlook on OpenID today.

Many site have their business model and valuations wrapped around the number of registered users that they maintain. OpenID does NOT make this model go away, it merely provides an easier way to begin a relationship with new users. It even makes it easier for the user to interact with a new site. The full set of benefits for offering support for OpenID is:

Faster and easier sign-up. Users get the choice of ID they prefer to use and can also choose how much information you get to see about them. You can still ask any necessary registration questions after they sign-up with their OpenID. Note also that field experiments have shown that the simplest possible registration process is 3 times more effective at eliciting sign-ups. OpenID naturally takes advantage of this fact and is the perfect complement to the highly effective lazy registration pattern. And don't forget that you still get to add the user to your local account database with all the information you care about, just in the context of their OpenID, meaning that you don't get to know their password or any other information they don't want to give you.
Better sign-in process and lifecycle. Users only need to remember one user ID and password and so will always have an easier time logging in. It also places the burden of password maintenance and account support (at least some of it) on the 3rd party OpenID provider. And since the OpenID standard makes sure that passwords are used in a secure way and never passed along directly to sites that offer OpenID logins, users feel safer continuing to use new sites. As I've explored in the past, OpenID also potentially allows enterprises to extend their single sign-on experience out across the Web to SaaS applications that support OpenID.
Consistent Web identity. There have been many discussions recently about namespaces as the new lock-in, particularly today's post from Chris Messina, and OpenID allows users take advantage of this to establish their own namespace consistently across multiple Web sites and indeed, the entire Web. When someone sees a given OpenID on a site, they'll know it's the exact same user they see on another site. Giving users the ability to unambiguously identify themselves authoritatively on your site will be seen as an increasingly important requirement of today's social Web.

To actually support OpenID, you'll need to implement the current specification or use one of the many available OpenID libraries that already does it for you. Support exists for all the common languages: C#, Java, Perl, PHP, Ruby, Python, and even Haskell. The bottom line, you'll be learning about digital signatures such as HMAC-SHA1 and extensible resource identifiers, or XRIs.

It should also be noted that the getting the user experience of OpenID right is essential for being successfully with it. Because most visitors will still be unfamiliar with it, a lot has been learned about how to offer the sign-up and sign-in experience in a way that is accessible and understandable. Early OpenID adopters often made the process too complicated or inexplicable to the user. A lot has been learned about how to do this correctly to enable the double digit usage rates we're seeing today on sites that are successful with OpenID. Probably the best presentation about effective OpenID user experiences is Brian Ellin's deck from this year's OpenID UX Summit, I encourage you to study it.

OAuth

Using a site through its user experience is now only one way in which the Web will interact with your service. If you're only offering a visual user experience and don't currently have an application programming interface, or API, you're behind the times. Open APIs are one of the most powerful new models for delivering services on the Web and they enable 3rd parties to integrate with and build on top of your product, creating all new services and mashups. This leverages something I refer to as Jakob's Law, which says that most of your traffic will ultimately be driven by external activities on other sites. Thus, more usage will often come in through an API than through the user interface. For example, Twitter famously has 10 times more usage through its API than through its Web experience, which I recently reconfirmed is still the case with Twitter's Alex Payne at Web 2.0 Expo earlier this month. It's also no accident that Twitter recently added support for OAuth.

However, the challenge with APIs is the same as with user experiences: How to let users access their data remotely in a safe way without having to give the 3rd party application their user ID and password for your site? Users might even feel comfortable giving their credentials for a Twitter application like Twitterific, but they certainly aren't going to do that for critical applications like e-mail, banking, or e-commerce. Enter OAuth, a method for allowing 3rd party applications to gain access to user data with their permission, but without having to give the credentials to that application. In other words, users can "authorize" 3rd party applications to access their data in your site, without giving that 3rd party application the user ID and credentials.

The benefits of securing your API with OAuth include:

Safe, secure 3rd party data access. Users will be able to give permission for specific apps to access their data on your site without having to share their user ID and password with the 3rd party.
Declarative control over 3rd party access. Users can also block the access of 3rd parties they previously authorized without having to have access to the 3rd party application. This gives them declarative control over who accesses their data and makes them more likely to use such applications.
Greater use of 3rd party applications drives local use. For many applications, handing out user IDs and password to 3rd parties is unacceptable to users. They will only use the sites they trust. Since the APIs often become the dominant channel through which usage occurs, meaning that 3rd party sites can often represent the bulk of interaction if appropriate security controls are in place and users feel safe, OAuth can drive increases in usage and traffic.

You can find the OAuth specification here but most will opt to use one of the many pre-built libraries that is already available and which greatly simplify implementation. The language options are much the same as with OpenID which is good since OAuth is seeing broader and broader adoption as its benefits become clear and it gains critical mass.

Reconciling OpenID and OAuth

Now, those paying close attention to this discussion might notice that OpenID and OAuth seem in direct opposition. OpenID allows users to employ remote 3rd party logins and OAuth tries hard to make the local ID safely usable externally, at least when it comes to access through the API. In reality, OpenID and OAuth are an excellent combined strategy and plenty of work is underway to make it possible to use OAuth with OpenID. Sites that support this combination will use the user's preferred login for both the user experience (UX) of the site as well as for the API, creating a seamless, consistent experience. The Step 2 project has begun to "combine the OpenID authentication and the OAuth authorization protocols" and has involvement with many of the major players in the space including Joe Smarr of Plaxo and David Recordon of Six Apart.

OpenID and OAuth represent major, scalable Web identity authentication and authorization systems that work with both a user experience as increasingly popular API access methods. They are both open standards and have the backing of many major players, particularly OpenID which has the backing (though only partial implementation) from Yahoo!, Microsoft, IBM, Google, Facebook, and other influential organizations.

Facebook Connect

Open Web advocates are probably wondering why I've included the proprietary Facebook Connect in this discussion of Web identity. That's because there is as of yet no major social identity system that is open and has broad support. Consequently, I will warn you that using Facebook identity is something that should be done carefully with eyes wide open for the lock-in issues and dependencies that can result. However, the value proposition is considerable. For example, Facebook claims that for many sites "2 out of 3 new registrations come via Facebook Connect, and those users have about 50% more engagement on sites". This alone will be worth it for a great many services, and certainly thousands of services have agreed.

By social identity I'm referring to not just a user's individual identity but also their social graph, or list of connections they've established online, as well as the communication that takes place between those contacts, typically an activity stream.

Social identity is fast becoming essential context for for online applications; it's not just enough to have you as a registered user since today's Web applications also need to know who your connections are in order to provide the services they offer for communication, collaboration, etc. Social identity systems allow you to login and then be able to refer to and access your social connections and communicate with them. This is far preferable to the old-school equivalent: Importing your contact lists and then inviting all of them to the new application you're using. The days when that was acceptable are over, if they ever were. Now you can just use your social identity and they can see what you're doing in their activity stream and decide whether or not to respond or participate.

For example, I use the video sharing site Vimeo quite often but many of my friends don't and don't even have an account there. But they can see what I'm doing on Vimeo with my Facebook Connect link that shows my videos in their Facebook activity streams when I post them. They can then decide if they want to view the video or otherwise interact with Vimeo.

Facebook Connect itself is a simple set of JavaScript libraries and XML markup that allows you to enable users to connect to their Facebook accounts from within your Web application. The capabilities and advantages of Facebook Connect are:

Single-click login. Users can log in with their Facebook identity and allow your site to access their Facebook profile.
More engagement. Use a user's personal interests to provide more relevant information and offer customized content from friends.
New distribution models. Users can share content and actions taken on your site with contacts back on Facebook through the activity stream and other social communication channels on Facebook.
Direct access to Facebook. Use the APIs that have allowed more than 700,000 developers to build tens of thousands of applications.

Getting started with Facebook Connect is relatively easy and the startup guide is straightforward. The integration is very platform-agnostic and requires little implementation for basic capabilities though it does use proprietary JavaScript libraries and XML markup. To get the most out of Facebook Connect will require deeper integration however, and this will create more dependencies on how Facebook manages social data and activity information. Fortunately, most good architects will be able to create a straightforward separation of concerns and one that will allow other social identity systems to be plugged in as they emerge, especially on the open side, such as Plaxo's Open Social Graph.

While many organizations remain uncomfortable delegating any aspect of user identity to external services, the reality is that online services are invariable stronger, more robust, and more attractive to users when they do. The effort is higher however, and there are more dependencies to manage but Web identity has definitively entered a near era. Proactive companies that take advantage of what is possible today will be poised to enjoy additional growth, higher user retention, and better customer relationships.