Enterprise Architecture

Can We REST For A Minute? 6 Lessons From The CRUD vs. Hypermedia Debates

Dion Hinchcliffe — Thu, 06 Aug 2009 12:31:00 GMT

Recently InfoQ did a good summary of the debates around the apparent (to some) limitations of REST when it comes to creating good Web services. At issue is that REST APIs seem to expose "CRUDy" services that fly in the face of years of good services design, particularly when they are just read/write interfaces instead of the richer, full REST architecture (more on what this is later.) The discussion was spurred by Arnon Rotem-Gal-Oz's assertion recently that CRUD is bad for REST, which in my opinion is close but not quite right.

In my view there appears to be two primary ways that REST are used today. One is via the aforementioned CRUD approach (where CRUD = Create, Read, Update, Delete) and HTTP's verbs (which in the same order as CRUD are POST, GET, PUT, DELETE) are used to manipulate a data resource on a Web server. This is the most basic, most common way REST is used by application designers (sometime even using these to replicate traditional RPC) and is the foundational request/response model in REST services. Unfortunately, it's also probably the least interesting or powerful way to apply the full REST architectural style.

For one thing, using only CRUD to manipulate data treats resources as loose, unintelligent bags of bits, seems too fine grained, and generally feels like client/server in a cheap disguise. Sure, it works extremely well with the protocol that runs the Web, HTTP, but no architect would consider this story so far good enough reason by itself to bend an entire application design around it. Things get much more interesting, however, when URIs start getting added to the mix. This happens in one of two ways and, depending on how URIs are applied, it's the key to the whole REST hypertext approach (aka HATEOAS, or "hypermedia as the engine of application state") which has worked so well on the Web for content. And now, with REST, we're doing it with data.

Data as Hypertext: Deeply Linked URIs and URIs for State Transitions

First of all, like the Web itself, REST works better when the data itself links to URIs of other resources that contain a more complete representation of a given piece of data (a line item summary might link to the full line item resource, for example). Deep linking, and not maintaining local copies of a representation is essential to prevent a variety of issues including data duplication and out-of-date copies.

Second, the very essence of REST is to make the states of an application explicit and addressable by URIs. The current state of the application state machine is represented by the URI that the client just operated on and the state representation that was returned. The client can change state simply by operating on the URI of the state it wants to move into, making that the new current state. A given state's full representation includes the links (arcs in the URI graph) to the other valid states that that can be reached from the present state. So in the visual below, we show a simple but complete interaction model that includes a single valid state transition. In short, as even Roy Fielding himself recently emphasized last year, REST APIs must be hypertext driven.

Click To Enlarge

The scenario above shows a complete and I hope canonical (albeit simple) interaction model that exhibits the full RESTful properties. The individual numbered steps of this scenario are described below:

Step 1: To use a well-designed REST API, only a single well-known URI needs to be published by the creator and referred to by the client. From this top level resource, all the resources and state transitions for the entire API and all its services can be found. A REST API is self-descriptive and therefore self-contained, with only minimal "out-of-band" documentation required. A simple HTTP GET at this URI will retrieve the API entry point.

Step 2: The API is examined and a service known as FooService is found, along with the URI for the entry point to the FooService API.

Step 3: The resource list for FooService is retrieved via GET. The URI came from the top-level API description examined in the previous step. In this example, a list of fooitems is retrieved, including possible state transitions.

Step 4: There is only one state transition, "reverse". This operation is chosen to be used in this scenario.

Step 5: A PUT (which means the resource will be modified) is issued to the URI described by the original resource. A back-end component (business logic) then reverses the name element of the resource (changing the state of the resource). A copy of the modified resource is also returned by the operation, making the operation efficient as well.

Step 6: Although a copy of the resource was returned from the state transition, the change is durable but we do another GET, this time just on the resource to see what the current state is.

Step 7: The state of the resource is examined and found to be changed by the state transition operation.

Lessons learned from the CRUD vs. Hypermedia debate

So, I think one of the biggest problems with REST is that there is no canonical description for how to use it. This is also one of its core strengths. There are many ways to apply REST to the problem at hand, and if all you need is CRUD, you are still doing basic REST and meeting your needs. No single description could do justice to the power and flexibility inherent in the model and I agree strongly with Sam Ruby that "REST isn’t an all or nothing proposition. One can get significant value from partial adoption."

But this discussion does highlight some very specific lessons learned that many REST practitioners don't often follow but are important to consider carefully in order to build services that are resilient, adaptable, and maintainable over the long term. Many of the benefits of REST aren't felt until later when you want to fix or change things. That you can do so easily is a testament to the potency of architecture of the Web, if only we can understand it clearly enough.

Here are my takeaways on what most REST adopters can and should do to get the most from their use of this increasingly popular architectural style:

Describe your REST API via a single, well-known, top-level resource at a permanent, public URI. This resource contains descriptions and URIs of all the services (by type) that are maintained by the REST API. This provides maximum flexibility for changes down the road since well-written client code will process the changes as normal operation and not require specific alteration for many kinds of modifications to the API.
Use business logic behind the server CRUD, return HTTP errors as appropriate. This one isn't as well known as it should be but REST services do not have to be dumb repositories of resources. They can do as much validation or processing as required for all POST, PUT operations. Don't accept data blindly from clients and return HTTP errors that make sense and describe the problem. Does this violate the self-describing aspect of REST interfaces? Not necessarily, the server imposes the valid state transition on the client and this is just another means of doing so.
Never hard-wire URIs into the client, always assume the resource will provide the current URI to related data as well as valid state transitions. Good REST interfaces will be easy to maintain for the server, but clients can take shortcuts, either to improve performance or to simply coding. Don't do this, just dereference URIs every time and you'll pick up and use modifications to the interface largely automatically and with no code changes.
CRUD isn't the only way to change a REST resource, invoking a state transition's URI can modify data implicitly. This is where REST can become very elegant and provide a self-describing interface and state model for changing data. Designing state models aren't what Web developers are particularly familiar with but it's a potent programming model that encourages good discipline and works for most types of applications. You can use CRUD where state transitions just don't make sense.
Versioning is best done outside of URIs. Version all day long but don't hardcode it into the URIs themselves. This can easily be done with content negotiation. You can also read Paul Sadauskas great explanation of how to do this this well.
The art of describing state transitions is still very much an art, study the latest conventions. There seems to be little consensus on how state transition URIs should be documented. Does one use the href tag or the uri tag? How does the resource convey exactly what the transition does in terms of altering state. This and other questions are all relatively unanswered still, though they are fairly obvious to those that examine a given REST resource. There is more consensus, for example, on how to pass parameters into state transitions or operations such as searches and queries. This is where URL Templates have made some headway and it's important to study them.

I've frequently described REST as a foundational element of Web-Oriented Architecture for both enterprise and Web applications and open data in particular. It's been thrilling to see REST become so popular in recent years, even if we have a long way to go yet to reach a wide understanding of what it is and how to use it. I hope these ideas are helpful and I'd love to hear your additions below.

10 Must-Know Topics For Software Architects In 2009

Dion Hinchcliffe — Tue, 17 Mar 2009 16:15:00 GMT

In the last year or so, after quite a lull, the software architecture business has gotten rather exciting again. We're finally seeing major new topics emerging into the early mainstream that are potential game-changers, while at the same time a few innovations that have been hovering in the margins of the industry are starting to break out in a big way.

The big changes: The hegemony of traditional 3 and 4-tier application models, heavyweight run-time platforms, and classical service-oriented architecture that has dominated for about a decade is now literally being torn asunder by a raft of new approaches for designing and architecting applications.

These might sound like incautious words but major changes are in the air and architects are reaching out for new solutions as they encounter novel new challenges in the field. As a consequence, these new advances either address increasingly well-understood shortcomings of existing approaches or add new capabilities that we haven't generally focused on before but are becoming increasingly important. A few examples of the latter include creating reusable platforms out of applications from the outset (the open API story) or cost-effectively creating architectures that can instantly support global distribution, hundreds of terabytes of data, and tens of millions of users. There are others that we'll explore throughout this post.

These innovations are hallmarks particularly of the largest systems being built today (which are running into unique challenges due to scale, performance, or feature set) though these software advances are also moving across the spectrum of software from everyday corporate systems and Internet applications to new mobile devices and beyond, such as the emerging space of social networking applications.

Mainstays of application architecture such as the relational database model, monolithic run-times, and even deterministic behavior are being challenged by non-relational systems, cloud computing, and new pull-based systems where consistency and even data integrity sometimes take a backseat to uptime and performance.

Let's also not forget about Web 2.0 approaches and design patterns which are becoming ever more established in online applications and enterprise architecture both. Social architectures, crowdsourcing, and open supply chains are becoming the norm in the latest software systems faster than expected in many cases. Unfortunately, as a result, the architectural expertise needed to effectively leverage these ideas is often far from abundant.

To try to get a handle on what's happening and to explore these emerging topics, I've been doing conference talks lately about the transformation of software architecture that we're beginning to see in so many quarters these days and generally finding consensus that the exciting days of architecture are back, if they ever left. Now it's up to us to begin the lengthy process of taking many of these ideas into our organizations and integrating them into our thought processes and architectural frameworks and bringing them to bear to solve problems and provide value. As one software architect came up and asked me recently, "How do I get my organization to understand what's happening out there?" This is an attempt at addressing that question.

Here's a list of the most important new areas that software architects should be conversant in and looking at in 2009:

10 Must-Know Topics for Software Architects in 2009

Cloud Computing. This one is easy to cite given the amount of attention we're seeing in the blogosphere and at conferences, never mind the (considerable) number of actual users of popular cloud services such as Amazon EC2. While the term doesn't have an exact definition, it covers the gamut of utility hosting to Platform-as-a-service (PaaS). I've covered cloud computing on ZDNet in detail before and broken down the vendor space recently as well. While the economics of cloud computing can be extremely compelling and there is undoubtedly a model that will fit your particular needs, cloud computing is also ground zero for the next generation of the famous OS platform wars. Walk carefully and prototype often to get early competency in an architectural advance that will almost certainly change a great deal about the software business in the near future.
Non-relational databases. Tony Bain over at Read/Write Web recently asked "Is The Relational Database Doomed?" While it's far too soon to declare the demise of the workhorse relational database that's the bedrock of so many application stacks, there a large number of promising alternatives emerging. Why get rid of the traditional relational database? Certain application designs can greatly benefit from the advantages of document or resource-centric storage approaches. Performance in particular can be much higher with non-relational databases; there are often surprisingly low ceilings to the scale of relational databases, even with clustering and grid computing. And then there is abstraction impedance, which not only can create a lot more overhead when programming but also hurts run-time performance by maintaining several different representations of the data at one time during a service request. Promising non-relational solutions include CouchDB, which I'm starting to see in more and more products, as well as Amazon SimpleDB, Drizzle (from the MySql folks), Mongo, and Scalaris. While many applications will continue to get along just fine with relational databases and object-relational mapping, this is the first time that mainstream database alternatives are readily available for those that are increasingly in need of them.
Next-generation distributed computing. An excellent story today in the New York Times about Hadoop provides a good backdrop on this subject: New distributed computing models are moving from the lab and becoming indispensable for providing otherwise difficult to harness computing power when connected to previously unthinkable quantities of data. While traditional request-response models that are the mainstay of network-oriented computing are important, so increasingly are effective ways to process the huge amount of data that are now common in modern software systems. Watch this video interview with Mark Risher and Jay Pujara at Yahoo that discusses how Hadoop "enables them to slice through billions of messages to isolate patterns and identify spammers. They can now create new queries and get results within minutes, for problems that took hours or were considered impossible with their previous approach." While Hadoop has considerable momentum, other similar offerings include the commercial GridGain and open source Disco and there are many others.
Web-Oriented Architecture (WOA). I've discussed Web-Oriented Architecture on this blog now for several years and my most complete write-up is here. In short, the premise is that RESTful architectures (and the architecture stack above and around it including data representation, security, integration, composition, and distribution) are a more natural, productive, and effective way to build increasingly open and federated network-based applications. The WOA debate has raged for a while now since it became a hot topic last year but the largest network on the world has cast its vote and WOA is the way that the Web is going by and large; WOA-based applications just align better to the way the network itself inherently works. In my opinion, it is a much better way to create service-oriented architecture for almost all requirements, resulting in more supple and resilient software that is less difficult and expensive to build and maintain. For enterprises considering the move to WOA, here is good overview I did a short while back about the issues and the evolution of SOA.
Mashups. David Linthicum wondered today in Infoworld where the mashups have gone, clarifying that he believed they had become integral to SOA and for delivering value in enterprise architecture. In reality, while mashups are extremely common in the consumer space, to the point that it's just an every day application development activity, the tools and concepts are just now ready for prime-time in business. I've previously called mashups one of the next major new application development models and that's just what's happened. Mashups were also prominent in my Enterprise Web 2.0 Predictions for 2009 (item #7). If you're not studying mashup techniques, Michael Ogrinz's Mashup Patterns is an excellent place to start studying how they impact software architecture .
Open Supply Chains via APIs. I find the term open APIs, which an increasing body of evidence shows are an extremely powerful model for cross-organization SOAs, to be confusing to the layperson so I've begun calling them "open supply chains." Opening up your business in a scalable, cost-effective manner as a platform for partners to build up on is one of the most powerful business models of the 21st century. However, there seems to be a large divide between native-Web DNA companies and traditional organizations in understanding how important this is (it's increasingly mandatory in order to compete online). All evidence so far points to this as one of the most important, though potentially difficult, things to get right in your architecture. Security, governance, scalability, and ease-of-consumption are all major subject areas and our enterprise architetures and SOAs must be ready for this business strategy as more and more organizations open up. Here's my recent "state of the union" on open APIs.
Dynamic Languages. Though dynamic languages have been popular on the Web since Javascript and Perl first arrived on the scene, it's only been recently that it's become acceptable to develop "real" software with them. .NET and Java are still extremely compelling (and common) platforms for writing and running application code but it's dynamic languages like Ruby, Python, PHP, and now Erlang that are getting all the attention these days. Why is this? As I explored in a detailed comparison a while back, a trade-off in run-time performance has generally been found to enable a large boost in productivity by virtue of what this lets dynamic languages accomplish. It also doesn't hurt that a lot of work has gone into newer dynamic languages to make them extremely Web-friendly, which is now one of the most common use cases for any programming language. Dynamic languages have architectural trade-offs of course, like any technology, though increasingly frameworks like Rails, CakePHP, and Grails are built on top of them which bring the latest best practices and design patterns, something that is not happening as frequently with older platforms. The tipping point has arrived however, and dynamic languages are beginning to take the center stage in a significant percentage of new projects. Software architects should be prepared.
Social computing. Developers and software architects are often uncomfortable with social computing aspect of software systems today but Reed's Law has unequivocally demonstrated that the value of social systems is generally much higher than non-social systems. Or you could just look at the many popular applications out there that are driven by their social behavior and derive their (often enormous) value from the participation it entails. Whether this is YouTube, Facebook, Twitter, or thousands of other social applications (business and consumer both), the lesson is clear: Social architecture is an important new layer in the application stack and it I've since made it two entire quadrants of my view of Web 2.0 in the enterprise as a consequence. A List Apart has a great introduction to The Elements of Social Architecture and I've identified some of the core patterns for this in my Enterprise 2.0 mnemonic, FLATNESSES. Fnding a high-value place for social computing in our enterprise architectures will be essential for modern software efforts.
Crowdsourcing and peer production architectures. Increasingly, the public network (the Web) has been used to enable potent open business models that are beginning to change the way we run our businesses and institutions. This started with open source software and has since moved to media and is now encroaching on a wide variety of industries. The models for doing this online require software architectures that can support this including architectural models for harnessing collective intelligence, moderating it, aggregating it, and protecting it and the users that provide it. As I wrote a couple of months ago in 50 Essential Strategies for Creating a Successful Web 2.0 Product, these architectures of participation create most of the value in the software systems that employ them. If you're not sure this is a software architecture issue, just look at Amazon's Mechanical Turk or CrowdSound, that latter which is a widget that allows even end-users to dynamically include crowdsourcing into their applications. You can also read John Tropea's new exploration of this topic for an application layer viewpoint.
New Application Models. The Semantic Web seems to be on the rise again and I've already covered Platform-as-a-service and mashups here, but in addition to these we are seeing entirely new application models cropping up in scale online. Whether these are Facebook applications, next-generation mobile apps (iPhone, Android, RIM, etc), OpenSocial or just the increasing prevalence of widgets and gadgets, the trend in the atomization of software (which was done still perhaps the best and most effectively so far in Unix) is reminding us that we still have new discoveries ahead of us. While these often seem trivial, aka applications as a feature, it's also increasingly clear that these are going to be here to stay and can provide considerable point value when they're designed correctly. Certainly for next-generation intranets and portals as well as the online "desktop", micro-applications which have to contend both with scale and with being useful and secure while embedded in other applications is increasingly on the radar. Know how they work, why they are so popular (there are tens upon tens of thousands of Facebook and OpenSocial applications alone) and learn how they can be used to provide real utility and every day value.

Any list of what is new and important in software architecture must be personal perspective so I invite you to add your own below in comments.

A Discussion with David Linthicum About Web-Oriented Architecture and SOA

Dion Hinchcliffe — Tue, 18 Nov 2008 17:22:00 GMT

I recently had the privilege of being on David Linthicum's excellent Real World SOA podcast show on Infoworld to talk about Web-Oriented Architecture (WOA), a topic that readers here know I've been exploring for a while now. David's one of the most respected names in enterprise architecture and SOA and so I enjoyed the opportunity to discuss with him what's happening to SOA as it meets the Web and begins to evolve in new and interesting directions. We had a lively conversation that covered the gamut from innovation and enterprise mashups to open Web APIs and cloud computing.

A full transcript with links is provided lower down in this post. You can also listen to it interactively with the Web widget below, or you can download the the entire mp3 file of the Real World SOA Podcast episode on WOA.

Note: I created this transcript from the mp3 file using the terrific CastingWords service which is a WOA application that offers its transcription service by building on top of Amazon's innovative Mechanical Turk API.

David Linthicum and Dion Hinchcliffe discuss WOA in September, 2008

Male Announcer: From the offices of InfoWorld, this is the SOA Report with David Linthicum. Covering everything that you need to know about service oriented architecture trends and strategies.

David Linthicum: Hey guys, it's September 8, 2008, this is the 'Service Oriented Architecture Report' and my name is Dave Linthicum. Our topic today, my conversation with Dion Hinchcliffe about Web oriented architecture.

[music]

Now lets hear from a sponsor.

Female Announcer: Whatever integration challenges your company faces BEA has a solution to match your needs. No need to change to match our solution, we support your choice. No agendas, integration your way.

To learn more about business integration go to bea.com/businessintegration and download the white paper, 'Business Integration and SOA, a Revolution in Business Agility'

BEA, delivering business innovation, real customers, real transformation, real results.

David: Welcome back to the Service Oriented Architecture Report podcast. My name is Dave Linthicum.

I've got a treat for you this week. I went ahead and bit the bullet and brought in a guest, and did an interview. And this time, my friend and counterpart, and we've been blogging back and forth for about two years on the whole global SOA and the emerging Web oriented architecture space, and that's Dion Hinchcliffe.

Dion's a Web 2.0 blogger over at ZDNet and has a very successful consulting organization called Hinchcliffe and Company. Dion and I got together and decided to spend some time and pontificate around this issue of Web-Oriented Architecture, which is all that and a bag of chips on the blogosphere right now.

So anyway, here's my conversation with Dion, and I'll catch you on the other side.

David: Dion, why don't you tell the listeners a bit about what you do and a bit about your firm and what you guys are working on currently.

Dion Hinchcliffe: Sure thing. Thanks Dave for having me on again.

I operate Hinchcliffe and Company as President and CTO. We are a Web 2.0 and SOA transformation firm. So we really can work with Fortune 500 companies and really look at the ways that technology is evolving in the 21st century and how to deeply embed that into their business and transfer the way that they deliver their products and services. And SOA, of course, has one of the top level organizing principals as a key part of that story.

I also blog for ZDNet, the Social Computing Magazine, and a few other places and operate Web 2.0 University as well. Which, we kind of help people understand all the things that are happening out there. There are a lot of new ideas, new concepts, business models, and ways that we have to embrace the marketplace and delivering value to our business. That's kind of where I come from.

David: Yeah, Dion and I first started sharing ideas a while ago, around extending service oriented architecture into the realm of the emerging Web. And this was something that was very new and cool at the time. And now its being embraced a little bit more and people are starting to figure out exactly where the value is and how their enterprise systems can work and play well with systems that are out there on the Internet and bring these services and bring this value into the enterprise.

And also, start taking enterprise systems that have been around for years, and enterprise processes that have been around for years, and even new processes, and starting to outsource them into this area of cloud computing. Or the ability to kind of put out a lot of the business processes, a lot of the information processes and kind of outsource it onto remotely hosted systems that are ultimately going to be a lot more agile and a lot more inexpensive to run.

So what is your research showing in that area Dion, around the whole cloud computing area and how that's changing the dynamics of enterprise architecture going forward?

Dion: Well, you probably know I had this concept a while back called Global SOA, stating that the Web is the world's largest network. It was eventually going to become an enormous resource for any business, a system much larger than any enterprise today. And that we had to learn how to connect our businesses to that and leverage the value that's out there.

Now, a few years ago there wasn't a lot of value out there that we could leverage in a SOA manner. The Web is primarily web pages. But, we've seen this larger transformation; SOA kind of went in one direction with interoperability and modular services. And, the Internet went another way with the same exact thing in terms of people wanting to connect their systems together.

But, it was really a business imperative to begin offering API's a few years ago. And now, it's almost rare for the modern Web product to appear without a well defined set of interfaces that we would call SOA if it was in the enterprise. And, very popular sites like Twitter get 10 times their usage with the API than they do from the regular user interface.

And this is the sort of thing that you would expect from a successful SOA in our enterprise. There's a lot of value being accrued from what we were building. One of the things that we've seen in our research, is that the web has kind of mapped out a way to... how to make these things very attractive, very consumable so that people want to use them, having valuable data and services.

And of course, Amazon, Google, Yahoo, eBay and all the major web properties now have extensive API divisions, which are growing very rapidly. Amazon, actually, recently reported that their total global web traffic across all their sites have now been eclipsed by their API's. And then, has been exceeded since the beginning of the year until now, about three times their total global web traffic comes into their API and not their websites.

That's an enormous return, and we would love to get those kinds of returns on our enterprises. So, that brings up this whole discussion around Web oriented architecture that kind of says, "Well, can we synthesize these fields?", because there was a lot of similar technology and similar goals, but different ways of going about it.

David: Yeah, it seems like Web oriented architecture is a very descriptive term to describe notions and concepts that we've been talking about for years. But, the cool thing about Web oriented architecture, the way I see it now, is that now we have the mechanisms and we also have the beliefs within the enterprise that this is a viable future.

This is a viable direction for their enterprise systems, the ability to take a lot of processes, and take a lot of services, and put them out on basically, the platform of the Web. And also, consume services and API's, as you mentioned, over the Web, into the enterprise. And basically, create this environment where the demarcation line between the enterprise and between the Web is starting to blur.

And a lot of the critical enterprise systems are going to be out on the Web, Web deployed. They're going to be delivered via API's from the Web. A lot of existing enterprise systems are going to expose their value out to Web delivered applications and applications that exist out there on the platform of the Web.

For example, platform to service, and software to service and even some of the enormous number of Web API's that are starting to emerge. And if you look at ProgrammableWeb.com, there are just hundreds of that that are being exposed. And I'm looking out there every week, and there's three or four that are showing up every day.

So it's an exciting time, and the fact that we're basically moving our concepts and our architectures out to an area of understanding or a platform, that just has a lot of potential and a lot of excitement going forward. Interaction and integration of social networking and the ability to kind of join enterprises together and get to this whole real time economy that we were looking to get to. So that a sale that occurs within an enterprise is understood by a system thousands of miles away, they're connected via the Internet.

It's just got to happen, and it's going to happen, in my opinion. And, it's going to take some good architectural forethought and some good visionaries within these enterprises to drive in that direction. What are your thoughts on that Dion?

Dion: You mentioned the security issues, and that comes up a lot, especially when I'm talking to my clients and they're always concerned about information being transmitted over the Internet. And they're very concerned about, a lot of their core business processes actually existing outside of their firewalls.

Either on a software as a service player, or now the whole emerging world of platforms as a service for actually building applications out there. And, building enterprise class applications out there where enterprise data is being bound to them. What kinds of things should they think about in terms of security as they start down this path?

David: So the scenario that everyone wants to enable, but is then concerned about security is: what if you had your local SOA and you've got all these services out on the Web that you don't have equivalent functionality for? It would be great to create a single application or automated business process that could bring these together.

And companies like Kapow, have great stories about how your SOA really isn't complete because you don't have all the data that you need to do your job. It's out there on the network. Understanding where that data flows inside the application and giving the credentials, the logging into a mash-up that has both inside outside services and saying, "I know where that data is going and I know its OK."

That's the scenario that everybody is worried about. We want to do this, we want to bring these things together and get the value, but we don't want to risk the business.

Dion: Did you look at OpenAjax, the initiative? And, I'm not a big fan of large industry initiatives designed by committee teams who really dilute their value and over complicate the offerings. But things like Smash, which is a part of OpenAjax, is designed to create internal lines of communications in the mashup that are secure and safe. Those are the kinds of answers that we're looking for.

David: I got one as we finish up here Dion. What are three predications for 2009 in terms of the world of Web-oriented architecture that you think we are going to see? And, I'll give you three from myself, so give me your three first.

Dion: Sure. I think one thing that we're going to see that's very, very interesting, and this is kind of the big idea of prediction, and that is the Semantic Web's coming back. We've seen this tremendous resurgence as we finally get some tools and we get a simpler approach.

Very much like WOA, its radically simple [Dion: with microformats and the latest tools] and yet it scales to the size of the Internet. It built the Web, and that's why it's such an important topic. But semantic Web is going to come back and it's going to affect a lot of this because, of course, this is all delivered over those technologies. So we're going to see a lot of interest next year in that, I think.

On top of that, I think we're going to see two things happening, and they're not going to be big radar until 2010, I think. But, one is a mash-up tools on the enterprise, matured to the point that almost anyone can build some level of functionality out of a SOA or Web oriented architecture. Composition is getting that easy, it's almost drag and drop. And, that, I think we're going to see the full maturity of the mashup tools to consume these things. And that's really how we get the value.

And I think for my last prediction, I think we're going to see the BPM world, and that's business process management, and BPO orchestration on all of those things start to reconcile themselves with this to. And, saying they right idea but they might need a change in focus. And we're going to see a lot of announcements, I think. And tools are going to support WOA to enable these orchestrations and business process development scenarios.

David: Yeah, I don't disagree with any of that, especially the Semantic Web. I've been a big advocate of the Semantic Web for a long time. I just couldn't get anybody to pay attention to it. But now, its seems to be starting to get some momentum again, which is great news.

My three predictions for 2009 around the Web oriented architecture space is, number one, the absolute, just explosion in the Web API world. I think that everybody's realizing that it really doesn't take a lot of time and effort. It just takes some planning and just "go do it" to get your existing information and your processes Web API enabled.

Whether its rest based services, or other types of API's. And, I just think that whole world is just going to blow up in 2009. I think that everybody's going to have an API. You are going to have huge API directories like they have in the programmable Web right now that are going to just explode.

You are going to have directories that are going to be propagated down into enterprise repositories automatically, and products to do that. You are going to have the ability for these applications to go out and find these API's and the back end systems to do the automatic updating of the API's.

You are going to have service oriented architecture governance principles, run time things, that are going to start to be more Web delivered around the notion of API's. And, I think, that whole world is just going to be all that in 2009 as people move from the Web as a visual paradigm to a non-visual paradigm as you mentioned in your summary.

Next, I think the whole platform as a service space is going to explode as well. I think that the economies of scale and just the enabling technology will be there in 2009. So, the platform as a service offering from Google and things from Coghead and things from Bungee Labs will start to just kind of take center stage in how people build and deploy enterprise applications.

And, I think, we are going to see a lot of call for visionary architects who are able to take their existing architecture and start moving bits and pieces, as they can, out on the platform of the Internet and run them. And basically, run huge parts of the enterprise without these costly data centers and ultimately be able to shut a lot of these things down.

Finally, I see that global BPM, this kind of goes to your point as well. I am just going to extend it a little bit, back in the days of Grand Central and back in the days of moving processes out to these shared environments, I think we are finally going to see some traction in that space. I think we are going to have SaaS delivered business process management systems and then I think we are going to have business process management systems within the enterprise.

They are going to be able to link to these global enterprise systems. So, you have the private processes and the public processes, and the ability to link to both and automate supply chains. And, automate supply chains and automate the event-driven economy in between these various enterprises and businesses out there and in between various countries.

And it just started moving to this global information exchange and global service exchange, which is going to take the automation on what we are doing today, quite frankly not very well, using very dysfunctional and static and fragile architectures, into something that is going to be globally managed, globally available. And, is going to operate at about a fraction of the cost of existing infrastructure today.

Dion: I have to agree with all those predictions as well. And, it is really hard to say. I think it is really the business side that seems to be having the most trouble thinking about doing this and understanding the implications and what the risks are for all of us right now. And technology, as usual, is right now, just evolving so rapidly in the Web services and the SOA space that I think our challenge is really trying to embrace it and make something of it.

I think companies that will be most successful over the next five years are going to be moving to these new business models. That's how, yeah very exciting time David and 2009 is just going to be big year for all of us.

David: That's great. Well listen, my guest today is Dion Hinchcliffe and Dion was nice enough to spend his Sunday night recording with me on the podcast, and I appreciate his insights into the world of Web oriented architecture or service oriented architecture and especially the global service-oriented architecture, which is really kind of the next destination for this stuff.

Well, I will talk to you guys in seven days and thank you very much.

Dion: Great, thanks for having me David. Have a good night.

David: Well, I hope you enjoyed the conversation with Dion, I sure did. Dion is doing some great research in the world of service oriented architecture now and meets the emerging lab and that is something we have talked about many times in this podcast. I hope to talk to Dion again at some point in the future and back in the podcast.

Certainly read my blog at Infoworld.com and you will see me reference Dion's work and you will see Dion reference my work, probably I am sure this podcast as well. Anyway, lots of stuff going on in the world of web oriented architecture as you heard in the conversation. I am going to monitor that area and keep reporting back to you in the blog and the podcast as to what's happening in emerging areas, new technology, new trends, new approaches, all that kind of stuff.

I think, it is probably one of the most exciting times to be in computing over the next three years. I think, the whole game is changing in terms of how we are going to build and deploy enterprise architecture in systems. Well, who is going to have components that are going to exist within the firewall?

I think, that's always been the case, but I do think there is opportunities to become much more efficient and much more agile using resources that we don't own or host, that don't live in our data center and happen to live out there in the world wide web.

And by the way, it happens to be a fraction of the cost of doing things within the enterprise, and it also happens to be more sexier and more cooler and much more agile and much more cost effective. And that is a very exciting thing, if you look at what we have done in computing in the past and where we are going right now. I think a huge transition is going to occur in a very short period of time.

Also a reminder, I am going to be doing the enterprise architecture virtual event, which will be held September 30 from 9:30 AM Eastern Time to six PM and my session will start at 10 AM on Eastern Standard Time. So, that is going to be a virtual event, and I am going to be virtually there. The agenda is on the website, you can find that at Http://virtualconferences.computerworld.com/enterprise_architecture/ and you can find the overview of my session in the event, and I hope you guys can make that.

I think that is going to be a huge trend going forward, as people instead of hopping on airplanes, I don't know if you travel these days, but it is not fun, are going to attend conferences virtually and I am going to be a virtual speaker, so I am looking forward to that.

OK guys, long podcast this week, but it was great to hear from Dion and always you can reach me, my email address david@linthicumgroup.com and please make time to read my blog at Infoworld.com Real World SOA. And, don't forget, next week I am going to be speaking at the Service-Oriented Architecture Executive Forum that is going to be held in New York City and that is going to be the Infoworld event.

I am doing the keynote I think on Tuesday, so make sure you make it up to New York City and come see me and come talk to me about what you are doing in the world of service oriented architecture. I always look forward to hearing from people who are actually practicing what they preach.

All right guys, until next week best of luck in building your service oriented architectures. I will talk to you in seven days.

The SOA world begins considering Web-Oriented Architecture (WOA) in earnest

Dion Hinchcliffe — Mon, 08 Sep 2008 12:25:00 GMT

It's been an interesting few months in the enterprise architecture space as we look at where service-oriented architecture (SOA) is headed this year. I wrote a detailed exploration of this on ZDNet last April and the discussion since then has only become deeper and more urgent. Getting general consensus on a new acronym is always a difficult thing to get widespread traction on, much less a new architectural approach. I'm not personally caught up on what we call this next generation of lightweight, Web-aligned SOA either, but WOA is the best name I've seen so far.

With the WOA discussion it's also been clear that the SOA industry -- vendors and practitioners alike -- are protecting their turf and looking a little skeptically at something that has the potential to change the center of gravity in the SOA business. I think there is actually little threat here; most of the top-down activities that SOA initiatives have been putting in place, such as governance and cross-functional business architecture alignment, are just as appropriate -- if not more so -- when it comes to making WOA successful. We'll talk a little bit more later about SOA products.

And let's be clear here: WOA is a really a sub-style of SOA that is actually highly complimentary. I personally believe we've collectively discovered that we've been spending the last few years on a course that just needs a healthy and appropriate re-adjustment, with the concepts in WOA helping us find a better way.

WOA clearly offers a number of advantages to those doing traditional SOA today. This includes considerably improved service consumption models that are less expensive and time consuming to use as well as unleashing the tremendous power of link architecture to drive information discovery, leverage, and analysis. The profound business implications of open APIs and cloud computing are growing clearer by the week. WOA also provides an excellent on-ramp to access the many powerful new online product distribution models currently available today. Distribution of SOA is a woefully neglected topic and WOA can bring an extensive set of techniques, from syndication to extraordinarily low-impedance Web services.

For its own part, the term WOA itself goes well beyond the simple reconciliation of Web services technology from a SOAP-based model to one based on REST. For this very important reason, WOA is not synonymous with REST. WOA encompasses all the architectural issues that are drive by the design of the World-Wide Web, an architecture, I will point out that has continued to refine itself including the rise of open Web APIs), prepackaged service consumption mini-applications (aka widgets or gadgets in the Web world, the advent of JSON, browser-based mashups, the recent resurgence of the Semantic Web, and much more. So, hopefully to put the REST/ROA vs. WOA debate to bed; REST remains an absolutely core architectural element, but WOA by definition encompasses the full architecture of the Web today.

The Web also clearly includes the browser and it's the browser itself that has driven many of recent innovations and trends in WOA, particularly mashups and the interest in JSON. And when it comes to the enterprise space, the reason that WOA has become such a significant topic is for the reason I gave in my most recent sum-up of WOA:

"It’s important to remember that no small system can sustain contact with a large system for very long without being fundamentally changed by it. This is what is happening with businesses (the small system, no matter how large) and the Web today (the big system.) The intrinsic nature of the Web is driving major changes in how we create network-based products and services and is inexorably turning us into Web-oriented businesses. Businesses that want to be successful on this network without understanding its fundamental nature and capabilities are only delaying the time it takes to reach the full potential the Web offers."

Unfortunately, technology always moves faster than businesses can adapt, and the situation hasn't gotten any better, quite the opposite. Fortunately, the on-ramp to WOA for most organizations should be far less traumatic that many systemic shifts in the past, such as the one from structured systems to object-orientation. Even more helpful for enterprises looking to start down the WOA route, many SOA products already offer support for core WOA capabilities -- especially around REST -- even though it's a far cry from what the potential could be. The REST support in Java EE 5 and Windows Communication Server (aka Indigo) are good examples.

Here are just some the things possible for companies that open up their data and functionality in a WOA model:

Some Business Scenarios Possible with WOA

Information in a SOA becomes crawlable, searchable, and discoverable. The WOA model for information architecture is very different than traditional SOA. A fully-formed WOA consists of enterprise data stored in millions of granular, deeply-linked network resources (XML, ATOM, text, images, documents, etc.) with addressable URLs. Hence this full Web of data is visible on the network and crawlable. One of the most amazing implications of WOA is that information network-enabled in this way can be found via corporate search engine, even if no application has yet been written to access the data in question. This is also a key aspect of a distant cousin of WOA, Enterprise 2.0 where leverage of existing, unexploited knowledge assets is one of the core benefits. The resulting URLs can then be fed into mashups, dashboards, and any other application that can consume links, which is a large percentage these days. Consequently, the information landscape fundamentally changes and is made much more consumable with WOA. The implications of this are hard to understate since search is one of the key capabilities that made the Web so successful. And yes, of course, there are also implications for security and multiple-levels of access. This will be where the value of IT comes in to resolve the issue in systematic, enterprise-wide manner.
Web widgets provide a new way to make SOA distributable and self-service. Users and developers can project information and functionality wherever they need it almost instantly using Web widgets, which are connected to the WOA resource landscape underneath them. Ideal for portals, dashboards, and enterprise mashups, widgets have become a key building block on the Web and the enterprise are very behind. Widgets are an important WOA story because they project consumption of Web resources via these small, portable applications. Widgets are harder to develop on traditional SOA based on WS-I Basic Profile (SOAP, WSDL, UDDI) because the browser can't directly process it without too much work. The consumption barrier is too high with traditional SOA and very low with WOA.
Open APIs that expose WOAs directly to partners. Instead of having a handful of integration points with a few partners, open Web APIs are showing how Web-facing WOA can offer up enormous opportunities to let partners onboard themselves, help themselves to a selected set of data, and integrate into business processes. This also puts the entire burden and cost of partner integration on the other end of the network, since the WOA is already developed, secure, and ready to use. 48% of CIOs said they wanted to open up their SOAs to trading partners last year and WOA is generally the best way to do that.
Enterprise mashups have the data they need to flourish. Any SOA practitioner will tell you about the tipping-point problem. This is how to get enough services so that you're interesting to the rest of the organization, and can actually solve their problems. These days it's getting easier and easier to WOA enable databases, data warehouses, and existing applications and making this happen is increasingly straightforward with a rapidly growing set of tools to do just that. Good examples of WOA enablers include MySql's xAware or SnapLogic that can then be used to build applications using enterprise mashup platforms such as JackBe, Serena's Mashup Composer, IBM's highly anticipated Lotus Mashups, or even the reliable old Yahoo! Pipes.
Get users involved directly with SOA. One powerful strength of the Web is its ability to enable ordinary people to richly engage with networks such as the Internet. This has worked extremely well with Web pages, but not with Web data, until recently. The rapid growth in popularity of Web widgets, dashboards, start pages, OpenSocial, Facebook apps, and other forms of portable, service-enabled application functionality has driven interest for the same things inside the enterprise. While our enterprise platforms are busy catching up to this reality, users are ready to start engaging directly with WOA and composing feeds, lightweight services, widgets, and social applications into meaningful business solutions. But only if we can keep up with what's happening and enable it.

I'll be the first to admit that most IT organizations would find many of the examples above somewhat distressing. The shift in control, the increase in openness, the different way of thinking about architecture, the countless security issues and governance concerns will likely prevent movement to WOA at a rapid pace for many businesses. However, I believe these scenarios offer too much competitive advantage for them not to be a major goal for IT organizations over the next 18-24 months. Worker productivity, innovation, growth, and even marketshare are going to be directly driven soon by whether a company has an open API or not, whether WOA services are in place to unleash data to workers, and so on.

Here are some other key statements from SOA practitioners and experts alike over the last few weeks:

ZDNet's Joe McKendrick had this to say in "WOA wins hands down in a SOA popularity contest":

"SOA may benefit from WOA (and Web 2.0 in general) because it enables business end users to see and experience online services via composite mashups and cloud computing. SOA could be sold as an internal cloud that provides online services inside the walls of the enterprise. In this regard, WOA makes SOA real to perplexed business users. Plus, enterprise SOA implementations may function as islands of integration that will eventually be assimilated into a larger WOA, while still retaining boundaries."

Also be sure to catch Joe's new "Everyone Loves Web-Oriented Architecture" for an irreverent and informative look at WOA. His comments from Steve Bjorg that WOA came before SOA are priceless.

Joe also cited the Information Week article on WOA by Roger Smith titled "A Simpler Approach to SOA", who reported:

"A growing number of companies are finding that lower-visibility Web-oriented architecture (WOA) developments, spawned through grassroots movements, are a better route to the service-oriented architecture. WOA, like SOA, is an architectural approach to system design, though WOA is resource-oriented rather than service-oriented. What's the difference? While the core SOA design unit is a reusable service that fulfills a distinct business function, resource-oriented services are more limited and data-focused."

Well-known SOA expert David Linthicum recently weighed in on Infoworld with:

"What's attractive about WOA is the fact that it's just sexier and easier to understand than SOA. Moreover, it incorporates many new other cool buzzwords such as cloud computing and mashups. I think what's most attractive is that it represents the movement of critical and core business processes from the datacenter to the cloud. This trend will continue, but it's going to be a slow migration over time, with some very visible short term successes, typically around outsourced infrastructure such as the new array of infrastructure services offered by Amazon."

David and I discussed this latter point last night in an upcoming SOA Expert Podcast episode and we agree: The biggest story for WOA will probably be its cross-firewall implications for API divisions that monetize SOA and provide scalable, self-service trade partnering capabilities, though the inside of the business will be a big story too. I'll update this post with a link as soon as that's out.

So we'll see how WOA fares over the next few months, but regardless of the success of the term (I suspect it will stick), the ideas behind WOA are going to remake IT and business over the next few years and I'll continue to cover it here. Please share your WOA thoughts in comments below.

Please read my popular What is WOA? article and 12 Things You Need To Know About REST and WOA for a deeper examination of what WOA is and why it's often a better way to do SOA.

12 Things You Should Know About REST and WOA

Dion Hinchcliffe — Tue, 08 Apr 2008 09:41:00 GMT

There have been a number of interesting tracts written lately about that increasingly popular topic in the world of SOA and Web services: REST. In particular, the one that is circulating around social bookmarking sites and SOA blogs the most in the last month is Stefan Tilkov's excellent Addressing Doubts about REST. The article tackles the continuing skepticism that SOA practioners have had about the integration approach that has become the dominant one on the Web and is now making significant inroads in the enterprise (more on my findings on enterprise adoption of REST here soon).

Key SOA Trend: As of March 2008, leading industry analysts -- such as Anne Thomas Manes -- are concluding that "SOA is not working in most organizations."

Compellingly, Stefan goes far beyond the simple and often misleading SOAP vs. REST debates and makes a number of excellent points about the REST approach ranging from encapsulation and transaction boundaries to documentation and tool support. But what struck me most is that these largely technical concerns, while very important, still don't strike directly to the heart of what makes REST, and Web-Oriented Architecture in general, so significant to the practice of effective large scale software integration and composition. Specifically, the more I look at working examples of large scale SOA on the Web, the more I'm aware of the fundamentally different mindset and approach that are used by those that have an urgent business need to achieve deep levels of integration between many customers and trading partners.

So what are the big differences between traditional SOAP-based, top-down SOA and lightweight, bottom-up WOA? In the end, it's as much architectural and philosophical as it is technical. I'll also be clear and note that while successful large-scale SOA on the Web tends to favor REST, REST drives many of the concepts described below, rather than promoting them explicitly. In other words, REST resides at the core of Web-Oriented Architecture, which in turn describes a set of related approaches for creating a robust and bustling network ecosystem of loosely cooperating entities that typically compete for consumption via "architecture of the fittest." Here are some of the key things we've learned over the last half-decade that REST has been used widely to build WOAs:

12 Things You Should Know About REST and WOA

REST posits an interconnected information ecosystem, not an isolated set of point Web services. REST services are (usually) XML resources that are deeply linked together using URIs (via connectors in REST parlance) into a tapestry thousands and thousands of other Web resources. This is the model used by the Web itself, which uses the same model (thousand and thousands of HTML resources deeply linked with URLs). The key concept here is that REST resources can be linked with other Web resources made by the same, or entirely different, providers. If you build a REST Web service and publish it, it's highly likely that in a short while you'll be a referenced resource in another REST service. While this sort of data transparency seems in direct opposition to widely cherished beliefs in the software development community around concepts like encapsulation and separation of implementation from interface (aka information hiding), it turns out that the "side effects" of this kind of transparent information architecture are many, varied, and usually highly desirable. The Web has taught us that publicly visible deep links are enormously important to system architecture, even as important as the data itself, enabling vital scenarios like discovery, search, analytics, transparency, participation, increased consumption, high levels of scalability, and well, robust ecosystems of participating components that can openly consume (and sometimes operate) on this data. As a final note, REST services can in fact still completely separate interface and implementation while at the same time prescribing a specific set of interaction scenarios.
"...the interconnected galaxy of data itself is now the central construct that is consumed and operated upon by network components."
A focus on Design for Consumption instead of Design for Integration. While some SOA traditionalists might disagree, there is a tendency to focus excessively on the imaginary integration point, or seams, of an SOA using the traditional WS-I Basic Profile world and I've had long conversations in the SOA community about tools for schema conversion, data translation and mapping, and other complicated scenarios to make two endpoints talk effectively. In this view, both sides of the conversation must have the same exact lens on how to approach the integration process, or agree to disagree. I've called this the "tyranny of SOAP's mustUnderstand flag" and this, combined with the fact that you almost always had to have the same programming language and Web service toolkit at the exact same revisions on each side of the conversation, it results in practice in surprisingly low levels of practical interoperability. Traditional SOA is designed, by intent, to diverge and fragment both because of the design of SOAP but also the proliferation of dozens of heavyweight WS-* standards that put a very heavy consumption tax on the conversation. The XML Schemas (or XSD) used in WSDL have also turned out to be a rather poor choice for meaningful descriptions of information that pass across integration points. This is in sharp relief to the world of REST and WOA where extremely simple standards ensure that whatever programming language and toolkit is being used, as long as it can process simple HTTP and XML, can interoperate quickly and easily while referencing the services API documentation. In other words, WOA enables integration between anything that can process the Web while SOA enables integration only between the (increasingly rarified) stacks standards and protocols that a given traditional Web service endpoint supports. Thus REST posits a world of integrating entities containing an almost infinite diversity in participants that couple well and scale best because of extreme simplicity and very low barriers to consumption. In contrast, there are only a handful of SOA toolkits that have the levels of sophistication to handle the fuller vision of heavyweight SOA, and unless you're using them, you can't come to their party. Finally, another way to look at this is that REST is near the top of the tolerance continuum and thus will always be significantly more open, inclusive, and egalitarian from a consumption perspective. If maximizing opportunities for integration is your goal, the right approach for you should be clear.
REST security is egalitarian and is as secure as the Web itself. Some theorists will raise the concern that using protocols such as HTTPS to secure REST is like using a single blunt instrument to solve a delicate and sophisticated set of problems. In practice, the protocol that has successfully protected the majority of e-commerce transactions on the Web is good enough for most applications. If different or more sophisticated means are needed, you can enable them as well, but it's clear that the large SOA practitioners on the Web are not adopting standards like WS-Security. For example, Amazon's popular S3 service uses simple, straightforward HMAC-SHA1 signatures to handle the authentication of each and every request to its REST API. Balancing security with ease of consumption requires a careful tightrope walk when it comes to successful software integration and the security solutions for Web services being using on a large scale today are not the ones we expected 5 years ago.
Service interaction directly by the client is a first class citizen in WOA. You might be reading this and thinking it's a no-brainer. But client consumption is a surprisingly slippery subject in the world of traditional SOA. For one thing, many SOA architects still refrain from thinking of the application client as a place where services are consumed directly, at least as a primary architectural concern. The client in this view can be the browser, mobile device, native application, or whatever is being used. However, the rise of rich user interfaces as well as the mashup software development model has driven the requirement for many Web services to be accessible directly from the client as a first order design concern. However, this is where traditional SOA has had significant issues. SOAP, the fundamental SOA protocol, does not have direct support in any of today's Web browsers making direct consumption problematic for even simple SOA services, and quite difficult for WS-* style services. Even worse, the latest new rich Internet application platforms such as Adobe's Flex, which are have become true software development environments in their own right , often have surprisingly poor support even for such important standards as XML. That's not to say that adapters, bridges, proxies, and other solutions can't be applied to existing SOAs to project them into the client erna. But all of these bring their own architectural tradeoffs and needless (and expensive) complexity including more layers of data mapping, translation, and run-time performance. Like other fundamental protocols such as RSS and ATOM, which are directly consumable by virtually all clients today, the very best SOAs make service consumption by the client a first class citizen and its services highly consumable in any rich Internet application, Web mashup, mobile device, or from wherever it needs to be accessed. This is key aspect of WOA's Design for Consumption, like Design for Manufacturing did for a generation of engineering processes and directly enables many important new scenarios we are looking for in software integration and composition today.
Service contracts are simpler and suppler in a REST model. Coming from a formal software engineering background, I myself was on the fence on whether the lack of a traditional interface contract actually inhibits the high volume consumption of REST services, which as we've discussed, is supposed to be one of its great strengths. This is one area where traditional SOA appears strong on its face, using WSDL to describe and elaborate on the precise nature of the methods, structures and data types being passed back over the integration point. While the topic of the impedance between most programming languages and service contract formats is beyond the scope of this discussion, suffice to say that we've learned over time that WSDL generally encourages tools to be far too finicky about a service contract and tends to create a brittleness that doesn't need to be there. I've written about minimal surface area dependencies for Web services before and it turns out that the everyday pragmatic consumption of REST is just not hindered by lack of machine readable contracts. For a variety of reasons, this tends to encourage a dependency on just the parts of the service begin used, and not the entire service. While the formal computer science crowd will have concerns about building reliable systems on top of services that change over time and lack formalized, machine consumable contracts, the reality is that in an environment that increasingly seems to be heading for a much higher number of informal services produced by a much higher number of sources, the lack of formal contracts is increasingly a feature. In fact, contract by example is sufficient for most applications, although mature offerings meant for transactional use do tend to have a formal API description, just not always machine readable. The early industry SOA assumption that most users of Web services would only consume them when aided by sophisticated tools has not been borne out on the Web. In fact, the world of ad hoc integration via mashups has further shown this not to be an issue. However, solutions such as WADL seem to be gaining currency when you absolutely need a contract for a REST service, though most developers I know using REST are more than content to just interact with the service itself or use a wrapper library that is provided by the service creator or the community that's grown around the SOA or API itself.

Figure 1: All Web services and REST resources have a contract, implicit or explicit.
REST strongly complements traditional SOA, if you must have it. Though increasingly, you don't have to have it. REST generally has much better consumption scenarios, is faster, more reliable, and more likely to be usable by those on the other end of the network conversation. Wrapping SOAP and other SOA-style services in REST is a workable solution, depending on what you're doing. Heavyweight service-orientation is at the bottom of the tolerance continuum and can make sense of a specific set of requirements, but chances are that REST will give you most flexibility, options, and uptake.
REST and WOA enables and does not violate the principles of service-orientation. Thomas Erl, one of the leading SOA thinkers in the industry, has identified eight principles of service-orientation that are generally agnostic of the technology used to implement a SOA while directly supporting the reported benefits of a workable service-oriented approach including easier interoperability, high levels of reuse, more flexibility in design, and so on. These principles include abstraction, loose-coupling, service-contract, reusability, autonomy, statelessness, discoverability, and composability. All SOA implementations tend to comply with or violate these principles to a varying degree either intentionally or unintentionally depending on their requirements and other vagaries. In this way, each SOA implementation has countless accumulated design decisions built into it that embody the architects', implementors', and vendors' net assumptions for the best way to realize the services that comprise that SOA. REST and WOA bring their own unique emphasis around what important in a service landscape, but critically, these do not violate a single of the essential architectural principles of service-orientation and often enables them unique and powerful ways. I'll explore these individually as I am able in upcoming posts since the statelessness and service-contract principles are very interesting areas for many SOA implementors to understand in a REST world.
Industry Perspective

Enterprise IT and SOA experts David Linthicum and Dana Gardner have recently weighed in on WOA on both Infoworld and ZDNet.
We have reached a possibly final state of deconstruction between data and function. I only say final since the Web is increasingly having the last word when it comes to the largest and most successful examples of just about any type of system you can describe and we don't see anything emerging beyond it. And Web has an intrinsic model that is exerting a network effect of its very own; if one builds something now that doesn't align closely with the grain of the Web then it will get largely sidelined until it is somehow woven into it. In other words, build a Web service that's not Web-oriented and chances are good it will stagnate. But build one that's Web-oriented and thousands of people will likely beat a path to your door (there are other success factors here of course, such as having best in class data). But there's a very big discussion lurking here with the essential idea is that we've nearly come full circle from the days of object-orientation where objects were code that was very tightly coupled to the data it operated upon. At the time, it was an architectural concept, not just for local information hiding. We moved from there to distributed stateful objects, then distributed stateless objects, then components, network services, and many other models. Services and code, however, tended to have the upper hand overall and mostly stood in front of the data or the database. But we've undergone a thorough inversion of this model because of the growth of Web architecture and the interconnected galaxy of data itself is now the central construct that is consumed and operated upon by network components (code running on servers and clients). This is a very different worldview that we have had in most of the traditional software industry, but the Web itself has essentially trumped the conversation and provided us with what appears to be the most workable model yet for the architecture of highly federated systems and composite applications. And this new lens is very Web-oriented.
REST drives WOA but WOA extends beyond REST. I realize that WOA isn't a fully accepted industry term yet, but I do favorite it to terms like resource-oriented architecture (ROA). WOA does indeed start with REST but also encompasses and intentionally extends into other, closely related models for designing and distributing composite systems and services. A good example is the rise of the Web widget as one of the newer and more interesting "component models" for enabling distribution and consumption of REST-based services. The Google Maps widget is one pre-eminent example of how WOA goes well beyond simple REST and describes a complete and integrated "package" for a WOA capability that offers an open API via Javascript which provides deep access to remote Web services. All this and it's also in a nice and clean browser-side API that even includes best-of-breed visual functionality. In this architectural worldview, you take a much broader perspective on opening up and offering services that provides delivery all the way to the "last inch" and gives developers data and functionality in a format that allows an SOA to be consumed in the simplest and lowest barrier fashion. The reward for Google has been one of the highest rates of uptake within 3rd party composite Web applications (aka mashups) of any API on the Web. The SOA model here is infused with the latest open Web-based user interface approaches to the degree that even a default presentation reference model, ready for production, is included as part of the SDK. That's as a complete, holistic, and pragmatic a view of SOA as you're likely to see since the most useful and productive consumption models in service interaction are emphasized while at the same time tightly aligning with the architectural and application model of the Web. This is what WOA at its finest can represent. Off the shelf WOA components like this are springing up and being used all over the Web (tens of thousands of them can be found on Widgetbox and the Google Gadgets directory), but are very hard to find in a traditional SOA environment.
REST is deeply infused into the fabric of the Web today. Not only is every single hosted Web page presently in existence already a read-only REST Web service (in REST parlance, transferring the representation of the state of the page via HTTP using the GET verb), but the latest and most influential Web standards, such as the highly regarded Atom Publishing Protocol, are inherently REST-based as well. Thus the overwhelming majority of pure data Web services on the Web today are REST-based, particularly the several hundred million RSS endpoints that are currently live right now. I've heard multiple times the story of how an enterprise switched from SOAP-based services to open syndication models for example, because many more tools support simple data pulls over HTTP, never mind the other advantages we've already seen above. We are just not seeing that sort of organic uptake and pervasive adoption with traditional SOA technologies. That is not only because of the aforementioned network effect but it also takes into account the very important lessons that we've learned from the Web. And one paramount lesson, as we'll see, is not controlling the other side of the conversation, which is one the last big pieces of the WOA picture. Which is....
REST enables an inversion of control that drives adoption and integration. This is somewhat similar to the inversion of control we see in things like dependency injection, in that the more direct control we give up over the integration process, the most integration we get because we've enabled the scenarios for it out in the "cloud" of the network. In a very similar way that the hyperlink itself -- and the URI in REST -- allows anyone external to the linked resource to connect information together, without a finger being lifted by the originating resource, the REST model allows what some call outside-in integration with the potential of almost entirely allowing integration to happen entirely external to the integrated system. While you might be thinking that surely our SOA approaches up until now have enabled this, the practice has been creating an escalating stairway of barriers to hurdle: You must process all protocols layered in the SOAP envelope to participate, you must have a contract in WSDL, there's a strong preference for information in XML, you should use the same programming language/platform as the service provider to avoid translation bugs, and so on. These and many other requirements impose a great deal of unnecessary control on both sides of the conversation but particularly on the consumption side. We seem to be learning that the very best models for integration impose as little control as possible. REST informs us that we must have a common representation of state, but it could be XML, or JSON, or images, or video. But beyond that, we are not constrained as long as the representation will fit over HTTP. And in this way, control over integration is inverted to the consumer of the service, who can engage in a thousand new scenarios not possible when all the aforementioned constraints are made. REST can set Web sites, businesses, applications, and every other silo you imagine as free as we know how to do it. As simple as possible, but no simpler and thus the network can integrate itself and we can achieve the advantages of pull instead of push, fluidity instead of impedance, a bazaar of consumption instead of a cathedral of integration.
REST and WOA can handle systems of arbitrary complexity and size. The systems built today from Amazon's Web services and many others show that hundreds of thousands of customers can integrate effectively and operate simultaneous on the Global SOA and run their businesses using REST and WOA. This is the "my Web site is bigger than your enterprise" realization that is making enterprises look hard at what's actually working on the greater Web. REST and WOA are not just ready for prime-time, they are prime-time.

There are literally dozens of models for building services to connect systems together. However, HTTP is at the core of many of the more promising ones, including REST. Here's an overview of the most common service models for SOA today.

Conclusion

However, many readers of this article are doubtless still wondering if REST and WOA are really the end-all, be-all for service-oriented architectures. For now, we're seeing it as one of the best available options despite a great deal of work yet to figure out how to apply it fully to the world of the enterprise. Is it ideal for every single type of application and scenario? Of course not. Your mileage will vary entirely depending on your requirements and your understanding of how REST deeply informs system architecture. However, it's increasingly emerging on the short list for those integrating systems of even the very largest size and complexity as well as down to the simplest and most nimble application. We've also learned a lot about the strengths and its weaknesses of this Web services model, however, as a fundamental part of the Web (since REST is nothing more than HTTP applied to data), REST along with WOA is the model that underpins many of the largest and most successful networks (and SOAs) in history.

Finally, I'll be delivering a complimentary Webinar on this subject matter next week, on Thursday, April 17th, to present the full scope of WOA and how it can be used to drive adoption, better business outcomes, and make SOA work in the enterprise. I do hope you attend.

What are your concerns? What do you think REST is capable or incapable of doing for your applications. Please share your story below in comments.

What Is WOA? It's The Future of Service-Oriented Architecture (SOA)

Dion Hinchcliffe — Wed, 27 Feb 2008 21:37:00 GMT

The need for businesses to open up their silos of information and internal capabilities to their internal customers has become an increasingly pressing issue as organizations strive to increase operational efficiencies and innovate more effectively with existing resources in the business and technical climate of early 2008. And in the last couple of years, as exposing uniquely powerful sets of data to online business partners has moved into the mainstream in the form of open Web APIs, opening up our IT systems across the Internet has become a competitive imperative as well. Unfortunately, despite two decades of experiments in heavyweight software engineering (the alphabet soup of EAI, SOA, ESB) for solving these types integration problems, we've seen relatively marginal improvements for most implementors despite heavy investments by businesses large and small. In short, integration between the systems running our business still isn't happening at the levels we need. However in the last several years, promising developments from the Web are pointing a way to a better model that seems to overcome many of the adoption and effectiveness issues of traditional SOA and is gaining wider adoption yearly (see sidebar below, right).

Most of us would agree that we still can't easily get access to the data and the systems we need to in order to get our daily work done. Workers still spend a great deal of time copying and pasting data between their various applications, data is batched and then exported and imported between IT systems around the world millions of times per day, and information just isn't getting to the places that we want it to without unacceptable amounts of manual labor. Even though Service-Oriented Architecture (SOA) initiatives around the world have the right goals, most efforts have fallen profoundly short of our desired levels of integration and improved business agility.

However, the news isn't all bad, the fascinating story is that there is a place today where the deep integration of our systems and information on a large scale has largely been solved and is a foregone conclusion in most cases. And that place is at the leading-edge of the World-Wide Web, sometimes referred to as Web 2.0. This success story has taken a while however and it's also managed to fly under the radar of most enterprise architects and IT vendors in the process. The left-hand turn that Web services took early on in the Internet story (circa 1999-2000) with SOAP, WSDL, UDDI, and WS-I Basic Profile turned out to be definitely not the right answer for the vast majority of integration scenarios (we'll see why below), despite the continued prevalence of these approaches today in most enterprises.

In contrast, the vast living laboratory of the Web has provided a singularly different answer, than has a fundamentally different focus though it remains a close cousin to traditional SOA. This much more Web-oriented approach is something that many have called Web-Oriented Architecture (WOA) and is based on the immense tensile strength of the World Wide Web itself and its underlying architectural fundamentals. And it's based on the basic concepts and rich outcomes that have made the Web far and away the largest open network on the planet as well as the largest SOA presently in existence. At the leading end of this is the Web mashups story with enterprise mashups being one of the major improvements to the IT landscape that WOA is heralding.

So to loosely paraphrase a famous line from history, I come to praise SOA, not to bury it. In the process I hope to explain WOA as simply as I can. This is still important because WOA just isn't standard fare yet for discussion in many IT circles while it's something that folks that build online services out on the globally scalable consumer Web increasingly take for granted. There are few vendors (though growing) that have lined up behind this pragmatic, effective, efficient, and highly popular approach, further limiting the body of formal knowledge and support available to practitioners who want to transplant this profoundly useful and simple way of connecting our systems together.

You may have noticed I've left "easy" out of my lists of adjectives describing WOA. And that's because I don't believe it's easy to set aside the last decade of evolving mindsets, habits, investments, tools, and skill sets for a better alternative, no matter how compelling or promising. And WOA is more than just a way of building Web services, it's also an ecosystem mindset and as such can at times be less accessible to non-system thinkers, particularly if said alternative is not well documented in the industry. And though significant headway has been made recently, particularly with an excellent spate of books such as Sum Ruby's superb RESTful Web Services, the grassroots, emergent nature of WOA has not lent itself well to extensive formal documentation. Despite the improvements in the available literature, I still find that a succinct, direct, and complete explanation of WOA is lacking and here's my attempt to boil it down to the essential principles. I then contrast it with traditional SOA so that the differences and similarities are clearly highlighted. Here's what I've come up with:

What is WOA? The Basic Tenets

Information in a WOA is represented in the form of resources on the network and are accessed and manipulated via the protocol specified in the URI, typically HTTP.
Every resource on the network can located via a globally unique address known as a Universal Resource Identifier or URI complying with RFC 3986.
Resources are manipulated by HTTP verbs (GET, PUT, POST, DELETE) using a technique known as Representational State Transfer or REST.
Manipulation of network resources is performed solely by components on the network (essentially browsers and other Web servers).
Access to resources must be layered and not require more than local knowledge of the network.
It is the responsibility of the components to understand the representations and valid state transitions of the resources they manipulate.
The service contract of WOA resources is implicit; it's the representation that is received.
WOA resources contain embedded URIs that build a larger network of granular representative state (i.e. order resources contain URLs to inventory resources).
WOA embodies Thomas Erl's essential Principles of SOA, though in often unexpected ways (such as having a contract, albeit implicit).

WOA Implementation Guidelines

The basic tenets above paints a picture of a galaxy of nearly infinite granular information resources integrated into a deeply interconnected set of dynamic connections that can be processed individually for a given task, in part (for integrated applications), or as a whole (such as enabling a comprehensive directory or search engine of all data and metadata.) In other words, the Web model provides a single, open, and unified information architecture that is consistent, easily consumed, extremely scalable, securable, very reusable, resilient, and highly federated. The Web itself is the single largest example of this and increasingly, enterprises are adapting their existing IT systems and legacy silos to this model, discovering the advantages of this ecosystem model for information resources and enterprise architecture.

Read an exploration of why SOA and Web 2.0 both reflect two aspects of a "timeless way of building software".

But the basic tenets previous list are at the architectural level. How does an SOA practitioner ensure they are implementing a WOA model as they build WOA Web services and resource-enable existing services and IT systems. Here are the key guidelines to be aware of:

Every WOA resource should have the same unambiguously and globally unique URI on the local (SOA) network as well as the World Wide Web.
In general, URIs should be descriptive of the resource to the extent possible. For example, http://domain.com/blogs/feeds/sruby.atom is strongly preferrable to http://domain.com/resources/12345678.
The type of resource representation (XHTML, XML, MP3, AVI, etc.) should be encoded in the URI itself. Using the .xml extension at the end of a URI is a common convention, for example.
A set of resources of a particular type should expose all known URIs in some manner (a WOA resource that provides granular, paged navigation for example) to enable linking, discovery, enumeration, browsing, and consumption in general.
Query string parameters are generally not considered part of the URI possibly excepting resources that represent algorithmic or functional outputs. Move query strings to the URI whenever possible.
Encourage URI self-reliance by limiting information that is communicated via HTTP headers when it can be moved to the URI. The Web cannot propagate header information, but links can.
Resources should link to related resources via embedded URIs instead of making local copies. This is the core of the hypermedia concept that makes the Web and WOA provide its unique capabilities.
WOA resources must careful to preserve idempotency for state transition consistency.

Read a write-up of how to design a loosely coupled and highly resilient WOA/Client with minimum dependencies and best practices.

The major differences between traditional SOA and WOA

Is WOA really the future of SOA?

Here are some key datapoints on WOA adoption and trends:

The majority of new Web services on the open Web (which is the largest SOA in existence) are now released in the form of either simple XML over HTTP or REST and not based on traditional SOA approaches. Source: API survey of ProgrammableWeb listings.
Creating any basic Web page automatically creates a simple read-only WOA Web service. Bonus points if you're using XHTML. This highlights an absolutely key fact: The Web is WOA and competing Web service models, even if superior, would have a very tough time gaining similar adoption (and have been trounced on the greater Web.)
The latest Web development platforms, such as Ruby on Rails, have already decided to deprecated SOAP and expose REST Web sevices for all apps by default.
Amazon famously tested the popularity of REST/WOA versus SOAP and the vast majority of customers (tens of thousands) chose the REST/WOA flavor.

One of the more helpful ways of understanding WOA is to see how it's different than SOA since there is considerable overlap between these two models of using the network to integrate, interoperate, and collaborate. While both approaches leverage HTTP, self-describing data formats such as XML, are concerned about the use of open standards, and can be used to build systems of arbitrary complexity, much of the similarity ends there. Here are some of the most significant contrasts between the two approaches:

SOAs tend to have a small and well-defined set of endpoints through which many types of data and data instances can pass. WOAs tend to have a very large and open-ended number of endpoints; one for each individual resource. Not an endpoint for each type of resource, but a URI-identified endpoint for each and every resource instance.
Traditional SOA builds a messaging layer above HTTP using SOAP and providing unique and sometimes prohibitive constraints to the Web developer, while WOA finds HTTP and related transfer mechanisms to be the ideal layer of abstraction for most applications.
SOA was designed from the top-down by vendors to be tool friendly, while WOA was emerged form the bottom up from the Web naturally and has the best support in simple procedural code and an XML parser.
SOA uses WS-Security and other sophisticated standards for security, while WOA tends to just use HTTPS.
SOA must contend with the vagaries of XML Schemas for service contracts, while WOA largely ignores the issue and lets Web services naturally represent whatever formats are desired.
Traditional SOA is fairly cumbersome to consume in the browser and in mashups while WOA is extremely easy to consume just about anywhere.

I should close by emphasizing that I enjoy and use traditional SOA technologies like SOAP, WSDL, and XSD frequently. But as more and more of the consumer Web moves to a more Web-oriented model, the evidence continues to mount that approaches based on WOA are easier to implement, scale better, and result in much greater uptake and usage scenarios. Traditional SOA is facing a crises of identity at this point, particularly given fairly lackluster results for most, and WOA may just be the prescription we need to make SOA deliver the robust outcomes that we were formerly expecting of it. Especially read the article I wrote last year (below in the reading section on eleven new ideas for SOA architects) to show the promise of and a new vision for user-controlled SOA and other aspects that WOA enables and that traditional SOA tends to constrain.

Other vital reading on the convergence and evolution of SOA, WOA, and Web 2.0:

Note: I reserve the next few days to clean up and revise this article. In the meantime, please share your SOA vs. WOA stories and opinions below.

Update: ZDNet's Joe McKendrick provides some coverage of these ideas in a new post.

Creating Open Web APIs: Exploring REST and WOA in Rails 2.0

Dion Hinchcliffe — Thu, 10 Jan 2008 17:27:00 GMT

In my recent 2008 predictions for the future of Web services and open APIs for enterprise applications, I said that we'd finally see a large scale movement to newer, lightweight Web-based models for opening up our software systems and integrating them together. In other words, heavyweight SOA has finally fallen out of favor and lightweight SOA -- sometimes known as Web-Oriented Architcture (WOA), is in.

However, this sea change has long since taken place on the Web and this year will see best practices in this area take another major step forward as we'll examine below. The recent convergence of the Web, SaaS, SOA, and other approaches has also made the boundaries between our architectures and systems increasingly intertwine and blur. As part of this evolution, we have also watched the gains that successful firms like Amazon and Facebook have made by opening up their products on the Web. And strategically, as an industry, we've begun to find it a lot smarter to think in terms of reusable, interconnected open platforms instead of single-play software applications. Along this journey, we have begun a major return to the roots of the globally linked structure of the Web.

The rest of this post consists of two sections. One conceptual and one technical:

The issues around and motivations for the move to new Web development platforms and architectures.
A tutorial and walkthrough of how to use Rails to rapidly implement Web apps with open REST APIs.

The next generation of Web development platforms and open APIs

The daily reinforcement and continuous growth in the fundamental power of HTTP and URL link structure, which is directly driving the Web's overall network effect, has starting giving rise to a new generation of software architects and product designers. This generation has grown up deeply influenced by it and they tend to think about the creation of software in novel, new, highly Web-oriented ways. Though the classical software industry has a long and proud heritage of its own around methodologies, architectural approaches, and design patterns -- proven in the crucible of real-world implementations of years past -- in this decade the Web has managed to exert its own unique, irresistible, and pervasive influence on virtually all aspects of producing software. For example, agile processes have been pushed to the limit and beyond by the forces imposed by the realities of the Perpetual Beta. And the scale of even average sized applications on the Web are now the largest we've ever seen. The absolute necessity of cost-effective operations and the marketplace requirements of embracing the new business models for Web 2.0 applications -- including advertising, user generated content, and rich user experiences -- have also changed the fundamental technical and commercial ground rules for success. As a whole, these changes have been driving a need for new software platforms that are explicitly designed to help us efficiently produce scalable, compelling online applications while also addressing the reality of modern-day Web apps.

Many of us who have to create the next generation of Web applications have been taking a hard look at the new platforms that have been created for the modern era of very large-scale networked software applications. And I'll be very clear here: While a great many of the old ideas and techniques in software development are as applicable today as they were ten years ago, there are also an whole new set of constraints and enablers for which we have to be very good at optimizing. As the Web begins its 2nd major wave of maturity -- and depending on who you listen to -- there is a considerably less tolerance for older, inefficient methods for developing Web applications; vigorous online competition for marketshare and increasingly online-savvy businesses have a much better sense of what is possible and how much it should cost and when it should be delivered.

These factors as a whole have pushed us into a new era of productivity-oriented platforms that started years ago with languages like Tcl and Perl and quickly moved on to Python, PHP, Ruby. Ultimately we ended up where we are today, with advanced, highly-efficient frameworks for these languages such as Ruby on Rails and CakePHP. These tools now let us create Web applications literally 10 to 20 times more efficiently than the general purpose language platforms of the 20th century, and with both traditional software engineering as well as new Web 2.0 best practices already built in. These improvements have only spurred what can only be called a "radical" movement in the software business, which started with open source software (the peer production of software) starting in the 90s, and where we have arrived, with a dramatic departure from the way we used to look at software languages and platforms in terms of how vertical a software development platform could be before it lost general appeal.

These new efficiency gains and vertical focus, however, are almost exclusively aimed at the twins goals of developer productivity and good design. These are both admirable and important goals since programmer time always been one of the leading costs in producing software. Software applications also spend most of their lifetimes in maintenance mode and clean application architectures from the outset can greatly facilitate updates and revisions. However, over the same time frame, the run-time efficiency of our programming environments, partially obscured by a little help from Moore's Law and Nielsen's Law, has been in a major decline. This has been largely intentional, when it comes to supporting improved developer productivity, or entirely unfortunate, such as the general failure of the software industry to figure out how to help software designers fully leverage the now ubiquitous generation of multi-core processors.

Out of all this there has grown a distinct and growing tension between the need to rapidly and inexpensively produce quality software and the requirement for it to scale cost-effectively to millions of users. The simple fact is, which you can readily see in the Hard Metrics diagram to the right, is that the previous generation of programming languages and platforms is up to 40x faster than what many of would prefer to use today to develop Web applications. Yet the more you go to the left on both diagrams, the more that programming platform becomes extremely expensive and time-consuming to develop with. Why is this? There are two primary reasons.

One is that the more popular, older programming languages tend to be relatively low level and general purpose and were designed for a different, older set of constraints. This has given us baggage that is often not very applicable to the modern Web-based world. Second, we've become very good at understanding the idioms and "syntactic sugar" that makes developers more productive for Web development and we've put that into the latest generation of programming languages and Web frameworks. Unfortunately, the combined newness of these new Web development platforms and their preference for coding time efficiency in favor of run-time efficiency has conspired to make the results they produce relatively slow and resource inefficient compared to what is potentially possible. Newness in this case is also a kind of performance tax since we just haven't had enough time learning how to make these new platforms perform well at run-time, similar to early versions of Java before the advent of the Just-In-Time (JIT) compiler. Fortunately, efforts like Ruby .NET have made some notable headway in this space recently, but are not commonplace yet.

The intent of the rest of this article is to explore the new release of Ruby on Rails 2.0 and examine it in the context of the trends above. The ultimate 10 million dollar question in the Web development platform arena is: Are the developer productivity benefits, including the embodiment of many current Web application best practices, that are conferred by new generation Web development platforms like Rails worth their cost in terms of operational efficiency? Increasingly, whether you're a corporate IT executive or a programmer at an Internet startup, you're going to be facing this difficult decision when you choose your target platform. Questions like "is it worth 5x-10x the programmer time to get run-time efficient software?", or "should I just increase the investment in more processor cores and bandwidth in the data center?" will keep you up at night. Making the wrong choice has potentially serious long-term consequences in terms of what it will ultimately take to maintain and operate your application. A programming platform's implications for operations are particularly pronounced since Web apps require more operational resources on the server-side the larger they grow, unlike traditional, installed stand-alone applications.

One way of thinking about the problem is that it's almost never a good idea to bet against significant improvements in computing and network bandwidth. So far we've not yet seen much to indicate that large, regular improvements won't continue for the foreseeable future. Another is assuming that a platform should be used in a slavishly monolithic fashion for an entire application. In fact, as an insightful interview with Alex Payne, a lead developer of one of most well-known Rails success stories, Twitter, shows it often makes sense to move the slowest parts of the app into something faster. This is such a common situation in software development that it's long been codified as the Alternate Hard/Soft Layers pattern. And while these two considerations alone will go a long way towards helping one decide which direction to take, one must also look to where the industry is going as a whole. The new productivity-oriented platforms are here to stay and adopting strategies to use their strengths effectively while being proactive in addressing their weaknesses, is the best route to success with 21st century Web applications.

Where's the interface? REST doesn't have a contract description language and essentially uses duck typing. Read about best practices for WOA/Client development.

So all of these issues form the lens through which we must look at the modern Web 2.0 applications arena. But let's take an actual look at what we're talking about here. How efficient can these new development platforms really be? And do they are actually encourage us down the right paths in term of modern best practices in the Web 2.0 era? Let's validate this by actually building, hands-on, an entire Web application using one of these new productivity-oriented programming platforms, specifically using the newly released Ruby on Rails 2.0. Those following along will need a little bit of technical skill, but you'll see that these new platforms are tremendously efficient from a developer perspective. In fact, we'll have an application up and running in literally a few minutes after you get your Rails 2.0 environment installed.

Building a WOA-compliant Web Application in Rails 2.0

We're about to get our hands on Rails 2.0 and build a complete data-driven Web application. But first we have to understand a little bit about REST and WOA since that's the "return to the roots of the Web" story I alluded to in the beginning. Nick Gall originally coined the term WOA, which he defines for us here. It's also called a resource-oriented architecture, but at the core of both conceptions is an approach called REST, which I've previously defined with specifics for those of you not familiar with it. But the key idea is that REST is just a way of using the fundamental protocol of the Web, the Hypertext Transport Protocol (HTTP), to exchange information with anyone else on the Web. REST treats the information on the Web as URL-addressable resources, which includes traditional Web pages but also pure data including XML, video, and audio. REST, which is really just a style of using HTTP, leverages the architecture of the World Wide Web in a natural, organic manner. In other words, REST is the best way we currently know of to open up our Web applications to the rest of the world, an approach I have called the Global SOA in the past.

In contrast to object-oriented models for software, or the procedural models used by traditional Web services such as SOAP, REST only uses four methods, those built into HTTP itself: GET, POST, PUT, and DELETE, which themselves operate on data resources located at URI endpoints located on Web servers holding the data (typically a relational database under the covers). Consequently, REST applications tend to have a much larger (and transparent) set of surface area dependencies directly on sets of addressable Web data instead of on bundles of procedural methods through which XML schema instances are passed.

Since platforms like Rails embody many of our latest ideas about how best to develop for the Web, it should come as little surprise that the principle creator of Rails, David Heinemeier Hansson, recently observed that the latest release of Rails consists mainly of "a slew of improvements to the RESTful lifestyle." One of the most remarkable things about Rails is how it pays more than lip service to this essential resource-oriented view of the Web. As you shall see, since open APIs are one of the hot topics in the Web applications business at the moment, it's nice to know that every Rails app automatically gets its very own RESTful API. So, let's see this for ourselves...

Building an Open Web API in Rails 2.0

Step 1: Getting on Rails 2.0

To explore developing a Rails 2.0 app and creating/using its open Web API, you'll need to install four pieces of software on your test computer.

Install the Ruby programming language. Version 1.8.6 is highly recommended. Here are links to the Ruby Windows installer, Mac OS 10.4 instructions. Ruby is already installed standard on 10.5 (Leopard).
Use the Ruby Gems updater from a command-line or terminal session to pull in Rails 2.0 from over the Internet:

gem install rails -y

You'll know you did this right if the output of the command:

rails --version is:

Rails 2.0.2 or higher. Warning: If you have an earlier version of Rails installed, it will be upgraded automatically.
Install a database of your choice. MySql or Postgres are recommended but even SQL Server or Oracle will work just fine, though you will probably have to install their Gems separately. Rails is for designing database-driven Web apps so make sure write down your user account and password for the database. Make sure you've created a named database instance. Don't worry about application tables for now, we'll have Rails take care of that for us later.
Connect the database to your Rails application. First we create a skeleton app and then we'll tell it about our database instance. Note: Getting the database connection information and credentials right after you install everything is often the hardest part about getting Rails up and running.

First, let's create the skeleton application we're going to be using for the rest of our work. Go to a local directory of your choice and type the following from a command line:

rails railsapp

This will lay down the entire application structure for a Rails app, including a built-in Web server, WEBrick.
Inside railsapp there will be a config directory with a file called database.yml. Open it in your favorite editor and fill out the development: section of the file with your database credentials and save it.
Start WEBrick in a separate command-line instance by typing the following, and be prepared to stop and start it occasionally as certain application changes are made:

cd railsapp
ruby script/server
Download cURL, a command line HTTP invoker. Put it in a path you can reach from your command line instance. We'll be using cURL to simulate a RESTful Web API client and invoke our Web application's REST API upon a data resource we've previously created via as a user the HTML interface we've built in Rails.

Now we're ready to start development and testing of our Rails 2.0 application. Keep WEBrick running in one command line window so you can see its debug output (it will show you all the HTTP requests that go back and forth), and have another command line in the railsapp directory ready to invoke various Rails commands.

Creating Our Open Web Application in Rails 2.0

For the purposes of this demonstrator, we're going to build a very simple employee tracking application in Rails. We're going to use the newer syntax in Rails for easy creation of a full Web application with employee record creation, viewing, updating, and deletion. Rails will even create the database tables for us, all the user interface screens (albeit they will be unstyled), unit tests, and even a complete RESTful API, our final end goal.

Astonishingly, we're going to create all of this using only two short commands at the command-line. You'll see why Rails is one of the most productive Web development platforms available and this step in particular shows some of the radical ease of use that Rails proponents (myself included) are consistently impressed with.

Step 1: Create the basic employee tracking application

Rails use a well-designed Model-View-Controller architecture and in our first command, were going to ask it to create all three items for us for our employee tracking database as well as matching unit tests and cross-platform database scripts. To keep it simple, we're only going to track two fields for the employee: their name and their extension. You are welcome to add additional fields but will have to deal with the additional fields in the steps below.

In the railsapp directory, type the following command:

ruby script/generate scaffold emp name:string extension:integer

You will see a lot of output showing the files that the Rails framework creates to handle the employee data we've just specified. The command itself invoked the generate facility of Rails to create a new scaffold for a model called 'emp' which has two fields, an employee name typed as a text string and an extension typed as an integer. A scaffold is an initial, working application skeleton with basic functionality including database persistence and a matching set of HTML forms for the CRUD.

Now, believe it or not, the employee tracking application is now mostly finished, the only thing we need to do it to update the database so that it has the schema for the employee records. You can use Rails' rake facility to get this done. This will require that you have correctly set up your database.yml file, and you will have to debug any connection issues to get this step to work. To migrate our employee model, named 'emp' to the database, type the following in the railsapp directory:

rake db:migrate

Now, you can run the application by going to the emps directory on the WEBrick instance. Note that emp as been pluralized automatically by rails, so our applications is located at the emps endpoint. To access our new Rails 2.0 Web app, point your browser to:

http://localhost:3000/emps

You should see the listing screen of the employee tracking database that looks like the one below:

Click on New emp to create a new employee, and enter the data as seen below (aside: Roy Fielding is co-inventor of HTTP and the person who created the original vision around REST), and click on Create.

The data entered is then transmitted from the browser to the server and stored in the back-end database. You can then view it, destroy it, or add more employees with the user interface that was generated for us by Rails.

We've now completed a simple but fully functional Rails application from beginning to end. But what we've come here to see is the fully RESTful open Web API that was created for us along the way. For this we'll need to use cURL to issue the API calls via HTTP to simulate another online program integrating live with our Web application.

Step 2a: Invoke the REST API to GET the employee resources

Now we're going to exercise all the HTTP verbs on our open Web API to see how it works. The diagram below shows the overall lifecycle of a REST-based resource using our emp example. The good news is that Rails automatically offers URL addressable resources for all the data in a Rails Web application. This access can be controlled and channeled as needed but it's open by default for whichever views already have visual access via HTML forms. This means Rails developers get a RESTful API for their applications simultaneously as they develop their user interface.

Let's go ahead use the REST API to pull the data for the employee that we added above. We'll use the handy HTTP utility, cURL, to interact with the Rails application via HTTP. Note that the URL we'll use now has the '.xml' extension added to it. This tells Rails that we're trying to access the XML representation of the resource instead of using the HTML user interface (in other words, we're playing the role of a program instead of a human user.)

curl http://localhost:3000/emps.xml

Enter the text above in a local command-line or shell with the cURL binary in the execution path. You should see output that looks similar to the following below. It's an XML representation of the employee data in list format, pulled fresh from the server via the REST API, with the emps tag as the enclosing list structure holding individual emp instances.

<?xml version="1.0" encoding="UTF-8"?>
<emps type="array">
  <emp>
    <created-at type="datetime">2008-01-11T01:02:53+01:00</created-at>
    <extension type="integer">1234</extension>
    <id type="integer">1</id>
    <name>Roy Fielding</name>
    <updated-at type="datetime">2008-01-11T01:02:53+01:00</updated-at>
  </emp>
</emps>

It's idiomatic in Rails to use the id attribute as the primarily key for application data. In fact, this convention is required for a lot of the magic in Rails to happen automatically and the rake migration way back in Step 1 already took care of adding this column to the database for us. That means we can use the id as the final addition of our employee resource URIs for updating, getting, and deleting individual employee resources.

Step 2b: Update an employee resource through the REST API via PUT

Let's go ahead and update Roy Fielding's phone extension through the REST API. Since we can tell from the employee list above that Roy's id is '1', we can use that to let the API know which record we'd like to update. You only have to send two parts of the resource in the API call, the id and the attributes we'd like to update.

Create a file called put.xml with the following contents:

<?xml version="1.0" encoding="UTF-8"?>
<emp>
  <extension type="integer">5678</extension>
  <id type="integer">1</id>
  <name>roy fielding</name>
</emp>

Invoke cURL with the following parameters to actually update the phone extension in the resource on the server (and consequently in the database.) The -H parameter sets the header so that Rails knows that an XML representation of the resource is being sent to it. -T makes the HTTP invocation a PUT operation, and the URL of the resource is http://localhost:3000/emps/1.xml where the number 1 corresponds to the id of the resource:

curl -v -H "Content-Type: application/xml; charset=utf-8" -T put.xml http://localhost:3000/emps/1.xml

Step 2c: Add an employee resource through the REST API via POST

Now we'll add a new employee to our application over the network using the REST API. This employee will be Tim Berners-Lee, so we'll create another XML file called post.xml that looks like the following:

<?xml version="1.0" encoding="UTF-8"?>
<emp>
  <extension type="integer">1212</extension>
  <name>Tim Berners-Lee</name>
</emp>

To send this via a POST operation through the REST API using cURL, issue the following on the command line. The --data-ascii parameter identifies the file to send via HTTP to our REST API. Because the resource does not yet exist, the URL is the base of the resource type, http://localhost:3000/emps. Rails conveniently returns the XML representation of the added resource so the id generated on the server for the newly added record can be obtained in the client without a second call to the server. Add Tim Berners-Lee to our employee tracking application via the API:

curl -v -H "Content-Type: application/xml; charset=utf-8" --data-ascii @post.xml http://localhost:3000/emps.xml

A browse of the employees list via cURL or the employee tracking apps Web forms will see that Tim Berners-Lee has now been added to the application, including the database, via the REST interface.

Step 2d: Delete an employee resource through the REST API via DELETE

Now we'll go ahead and remove Roy Fielding from the database using our REST API. This process is straightforward and uses the HTTP verb DELETE. You can issue this via cURL using the following command:

curl --request DELETE http://localhost:3000/emps/1.xml

You can now verify through the employee tracking Web forms that Roy Fielding's employee record has been permanently removed from the database.

Conclusion

We've seen how Rails 2.0 makes it enormously simple to create a database-driven Web application, expose it via a REST API, and manipulate it via a REST-capable client in a clean, no-nonsense manner. Developing similar capability in C++, Java, or .NET environments is currently much more difficult. What you see above however, is only the beginning; Rails 2.0 has added a lot of other support for more sophisticated uses of REST and HTTP. I'll cover these in one of my upcoming posts as soon as I am able. The key point here is that the next generation of Web application platforms puts almost staggering amounts of power in the hands of the average Web developer while providing powerful capabilities like properly formed REST APIs automatically. This further puts the latest best practices for Web apps into places it wouldn't otherwise happen. Open APIs will help power the next generation of online success stories and for this and other reasons, Rails should be on the short list for those considering new Web development efforts. That is, only if they are prepared to do what's necessary to address Ruby's and Rails' shortcomings in run-time performance.

Still trying to exactly understand why Rails is such a compelling option? Read an analysis of why platforms like Rails are a major improvement over previous generations of Web application platforms.

If you have any trouble getting the code to work, please contact me at dion@hinchcliffeandco.com.

The state of RIA moves forward: Ajax, Silverlight, and JavaFX

Dion Hinchcliffe — Sun, 23 Dec 2007 15:33:00 GMT

I've just returned from Europe after delivering our new three day masterclass on Web 2.0 combined with the very latest in Rich Internet Application techniques for Web application development. We tried to create a highly innovative learning event that covered the latest in Internet design patterns and business models while at the same time tying tightly it with execution at the architecture and technology level. After covering Ajax fundamentals and then adding in an injection of the very latest cutting edge RIA technologies from Microsoft and Sun, I believe we delivered one of the most leading-edge, fast-moving, and educational three days of high concept and hands-on Web application development.

Delivering on this vision required a serious review of the latest alpha and beta code drops for Silverlight 1.1/2.0 and JavaFX as well as a deep dive into ASP .NET Ajax. We found that these RIA platforms and frameworks are really starting to get ready for prime time even in their "under construction" state. The capabilities that are germinating in these new platforms are fascinating and are clearly poised to move the RIA industry state-of-the-art forward significantly. This is the story of what we learned along the way...

Lately, we've been struck by the challenge in the magnitude of the diversity in knowledge that must be acquired and mastered before one can deliver online offerings that are successful, much less highly competitive. The Web is currently the world's the best place to create and offer an online software product and lack of potent RIA approaches were one of the key missing ingredients that held back the potential of genuine online software applications. Furthermore, the real upside potential of the Web is scale and more scale to a readily accessible audience of many hundreds of millions of potential users. The delivery models are now just about here while the audience and demand continues to grow (SaaS, or Software-as-a-service, is on target to be the dominant software model by 2011/2012.)

The challenge currently is creating RIA applications that can deliver in terms of a competitive user experience while still enabling the other key aspects of Web apps that drive their success. These key aspects including ensuring a strong network effect, turning the Web app into an reusable 3rd party platform via open APIs, avoiding the various standard issues with RIAs (loss of SEO, poor GUI conventions, significant reductions in page views, etc.) and last but far from least, a workable long-term business model beyond tons of unmonetized traffic.

The masterclass itself included a half-day of heavy duty Web 2.0 principles and then 2.25 days of intensive RIA study. The Web 2.0 portion focused on the main principles as originally espoused by O'Reilly Media when they defined the term and which also forms of the backbone of our popular strategic Web 2.0 University courses. We then consistently tied back the strategic principles to the discussions of the various RIA approaches (Ajax, Silverlight, and JavaFX), such as discussing widget delivery models for RIAs and organizing Web applications around a true Web-Oriented Architecture (WOA) with granular URLs/URIs and creating an applications public and private APIs in REST or JSON form. In formal Web 2.0 terms, this is the Lightweight Programming and Business Models principle, which can really help technical folks begin to map the overarching ideas to specific patterns and practices. We also had a good group of experienced developers as students, most with some RIA experience, which allowed us to really kick the tires on these technologies. Finally, we also said we'd create RIA scorecard at the end of the course to see how capable the class as a whole found these various RIA technologies for building Web 2.0 apps. So far so good...

Web-Oriented Architecture (WOA) in support of RIA architectures was a fundamental concept that we revisited repeatedly in the masterclass. Read an intro to WOA as well as our conception of WOA/Client.

The four RIA models that we covered and compared in-depth were 1) "bare metal" Ajax (without the assistance of a framework or library, 2) ASP .NET Ajax, 3) Silverlight 1.0 and 1.1/2.0, and 4) Sun's JavaFX. Given that AjaxPatterns.org currently lists over 200 Ajax frameworks, picking the Ajax framework for the course came down to the practicality of time and we selected ASP .NET Ajax from Microsoft since we were already going to have the development tools in place for Silverlight, which are almost identical to the ones used for ASP .NET Ajax. Some of you will ask why we didn't cover Flex or Flash, and the answer is mostly that we didn't have a request for it in this course. We will be adding a Flex and OpenLaszlo section shortly to this and bring it out to four total days that will cover the full gamut of the major ways to design and build an RIA.

Comparing and Contrasting Pure Ajax, ASP .NET Ajax, Siliverlight and JavaFx

Pure Ajax
Not for the faint of heart, building a Rich Internet Application with plain nuts-and-bolts Ajax requires fairly good working knowledge of a slew of browser technologies including the Document Object Model, JavaScript, DHTML techniques, XmlHttpRequest, timer management, CSS, and Web services/SOA/WOA. While the most compact and high performance result can come out of this, most Web developers will be looking for at least some assistance to elide browser differences, provide a component model and useful components, and handle plumbing, infrastructure, and assist in debugging, which is still in a largely deplorable state in browsers, despite advances such as Firebug. Pure Ajax has a lot of draw however, since it's based on 100% open standards and does not tie you to anyone else's code or to any vendors or organizations (other than standards bodies.) We had everyone build a "bare metal" Ajax application based on the Google Suggest concept to see what it's like to develop an RIA using only the capabilities that are built into every common browser today, and no external code, utilities, or libraries. Most attendees felt this model was going to work best for embedded apps but that it required a lot of effort.

ASP .NET Ajax
A powerhouse of an Ajax framework that I've been writing about for a few years now, this framework brings a distinctively Microsoft flavor to the Ajax arena. Although ASP .NET Ajax has a competent set of controls and even a full Javascript base class library, the strength of this framework is that it's designed to solve some of the tougher issues in building dynamic applications that are fed by server-side APIs. The UpdatePanel is the "crown jewel" of the ASP .NET Ajax framework (a good overview here) that is so compelling that it even has its own book on the subject. The essential job of the UpdatePanel? Specifically, handling the complexity of managing the updates of the visual elements of an Ajax application that are bound to remote Web services. However, despite the rich functionality designed specifically to simplify the Ajax development process, the GUI editor for ASP .NET Ajax in Visual Studio 2008 is somewhat confusing for the new user and is one of the minor disappointments in an otherwise excellent Ajax product. ASP .NET Ajax is fairly egalitarian in terms of browser support and the applications it generates run on most major browsers. For the masterclass, we did an in-depth overview of the platform and built an application that used the UpdatePanel.

Silverlight 1.0 and 1.1/2.0
Microsoft has been working for years on a Flash competitor that will provide developers with a large set of options for developing capable and compelling RIAs. While Siliverlight 1.0 only had basic capability in terms of simple animation and media support using Javascript as the primary scriptiong language, Siliverlight 1.1/2.0 (now officially version numbered as 2.0 a few weeks ago according to Tim Sneath) is a completely different animal. Silverlight 2.0 offers a complete .NET common language run-time in the browser including managed versions of Javascript and Python that will compile to binary on client and run extremely quickly. Obviously, by supporting Python (and Ruby as well, though not in the current alpha distribution) in the client, Silverlight's CLR in the browser now also support the Dynamic Language Runtime (DLR), making Silverlight have the richest support for RIA client-side languages currently available. Of course, sporting a lightweight version of .NET and its libraries comes at some cost, particularly download and installation time, but it's surprising at how much they fit into a relatively small footprint that comes in at only a few megabytes. Combine that will some of the neatest RIA apps I've seen (visit a few of these 50 different Silverlight demo apps), and Silverlight is well positioned to offer high performance RIAs that provides the broadest set of developer choices as well as potential creative inputs. Because it will ultimately support so many languages, libraries and code can be reused from a much broader source than has been possible before in RIAs. Silverlight also has incredible tool support with nearly seamless integration fo Visual Studio 2008 for developers and Expression Blend 2 for designers, which can operate on the same files depending on what you work on: the code or the creative parts of a Silverlight application. In the course, students used Visual Studio 2008 and Expression Blend 2 December Beta to create controls and applications.

JavaFX
Far from being left out of the emerging RIA story, Sun has created a very interesting new entry in the RIA space. Designed to leverage the full breadth and depth of the extremely mature and robust Java platform, JavaFX is a scripting language that doubles as a declarative programming model. With the express goal of making it significantly easier to create Rich Internet Applications than it is now with current Java technologies, JavaFX offers some serious productivity-oriented features including: A highly efficient Model-View-Controller (MVC) data binding construct in the scripting language itself, declarative event triggers for assertions and CRUD, and even some cutting edge features such as extents (a notation to let you see all class instances of a certain type) and other mechanisms that will give one some concerns about the sacrifice of long-term code maintenance to the altar of code efficiency, but it's a pretty well thought-out model. You can see some of the JavaFX demo applications here. For the masterclass, we examined an existing JavaFX application and explored how easy it was to declaratively modify the view of the RIA by changing the data model as well as add new features, triggers, and data bindings.

While highly efficient Web development platforms have emerged in recent years -- such as the rise of Ruby on Rails -- to show how much improvement there is to make to our often too general purpose programming environments, it's fascinating to see that the latest approaches, particularly Silverlight and JavaFX have explicit strategies that both focus on productivity-orientation as well as leveraging the vast existing sets of code, tools, and talent. Declarative approaches are the clearly favored here too, and though procedural coding is also fully supported in ASP .NET Ajax, Silverlight, and JavaFX, much of what RIAs need in terms of design fits extremely well with the declarative model that harkens all the way back to HTML, which forms the pure Ajax duo of XHTML (declarative structure) and JavaScript (application behavior.)

All in all, the state of the RIA is clearly getting ready to take a major stride forward in 2008 while the options for design Web 2.0 applications with RIA technologies continues to expand significantly. The RIA scorecard that the inaugural class created showed there all four options have strengths in different areas with Silverlight currently taking the highest overall score.

Note: For those of you that would like to hold a Web 2.0 Apps RIA Masterclass at your facility or attend one our public deliveries in 2008, please contact us.

Help form a picture of the state of the RIA industry

Please take 60 seconds of your time and participate in the RIA survey below so we can get a sense of where things are going in the RIA industry. Only participants (and the subscribers to the Hinchcliffe Advisory) will get a copy of the data, so it's worth your time, plus you can get a free subscription. Survey closes on January 31st, 2008 and the results will be issued in February.

Eleven Emerging Ideas for SOA Architects in 2007

Dion Hinchcliffe — Sat, 20 Jan 2007 15:29:00 GMT

As I highlighted recently on ZDNet, 48% of CIOs will be looking to actually start using their SOAs to connect to external partners this year. Unfortunately, we've been building landscapes of Web services for quite a few years now and for many, the tipping point for SOA adoption seems as elusive as ever. While trying to understand why this is, one common explanation I offer is that the A in SOA is often missing. When you ask server-side developers in a given organization what they are developing, they usually say Web services. When you talk to architects in the same organization, they usually say they are building SOAs.

This highlights a discernible and common disconnect between the tactical reality of delivering applications on deadline and the strategic goal of delivering carefully crafted services that are generic enough to be reusable, tested with all the technology stacks in the organization, and meet all other criteria of having well architected enterprise services.

This is where the World Wide Web continues to teach us effective techniques for service consumption and adoption. Amazon has tens of thousands of consumers of its various and sundry Web services that range from e-commerce to the compelling S3 storage platform. And they're making money doing it as well. The rise of mashups too has shown how easily that simple, composable services can be made into workable browser-based composite applications. All of these has given us the conception of Web-Oriented Architecture (WOA), which I've been writing about here on this blog for a while now. This is using the basic Web formats and protocols such as HTTP, XML, REST, and JSON as the "Unix Pipe of the Web" -- to quote a colorful phrase of Ray Ozzie's -- as the fundamental glue between systems. This allows widgets, Ajax applications, and mashups to be wired together so quickly it can almost be done in real-time with the latest tools.

Finally, we have Web 2.0 (most recent formal definition here), a way of leveraging the fundamental strengths of the Web to turn applications into platforms, exploit the potency of network effects, and otherwise take advantage of networks as true software platforms in their own right. However, the rules of networked platforms are very different than the ones we are familiar with on the computing-oriented platforms we know traditionally such as operating systems like Windows, Linux, and the application stacks that sit upon them. As it turns out, being a fundamentally communication-oriented platform, our networks impose a whole new set of rules for success that we are just now finally beginning to understand well. Surprisingly, among these, is the recent realization embodied by Reed's Law, which states that the instrinsic value of a network is much, much higher if the network is used in a social manner. Thus, in some important way, social networks tend to more fully leverage the value of networked applications and services.

Steeped in formal standards, byzantine product stacks, and software engineering principles, these are strange ideas for SOA architects to accept, much less embrace. Then there is the matter of usefully applying these ideas to create an effective service-oriented architecture that can be easily consumed by internal and external customers, and indeed, is preferred to use instead of reinventing the wheel. For the truth is, if the services most of us are building now were so much better than letting development projects just build it themselves, they would be beating a path to the nearest internal SOA representative to save themselves the cost and time. And while that is happening in some cases, SOA adoption studies and anecdotal evidence tells us it's just not happening enough.

So, in the spirit of lessons learned and to incorporate our most recent understanding of what works and what doesn't, I thought I'd put together a suitably provocative list of what SOA architects should be seriously thinking about in 2007:

Eleven Emerging Ideas for SOA Architects in 2007

Making services consumable in the browser. Increasingly, the common Web browser is the place where meaningful service integration is taking place. Because of this, building services that aren't easily consumable in the browser can be a death knell for the service because that puts its consumers in the business of building and maintaining adapters or using a Javascript SOAP stack -- if you can find one -- before the service can be used and measurable work accomplished. Ultimately, non-browser friendliness greatly reduces possible consumption scenarios for SOAs as we'll see in some of the points below. This doesn't' mean throw away your WS-* services. But it does mean you should automatically offer a REST or JSON version as well.

Considering syndication over "service-izing." The browser is an important consumption point but so too are the growing syndication ecosystems of which the blogosphere is the largest example. More and more tools are willing to consume RSS and ATOM, often in preference to SOAP, including the forthcoming version of Vista where syndication-friendliness is a core value. Carefully consider offering your services in RSS form or even ATOM, which has a two-way REST model. This will further increase consumption scenarios and therefore adoption. Content syndication is growing into a very potent force inside and outside the enterprise and plugging an SOA -- strategically or tactically -- into one of these ecosystems has terrific upside potential. Not every SOA service can or should be converted to a syndication model, but if you aren't considering this option with each service you create, you should be; there are tens of millions of RSS feeds available today, starting from zero in the beginning of 2003. How many SOAP services presently exist worldwide? Only a tiny, tiny fraction of this and there are good reasons for it.

Deeply embracing URI addressability. Of all the things in this list, this might be the most important one. The hyperlink is the fundamental unit of thought on the Web and it should be in your service designs and (hopefully granular) schemas as well. Giving each discrete piece of information, every service, and all content a globally addressable URI instantly gives a service, and the data it carries back and forth across its interface, access to countless new consumption and reuse scenarios. The most important of these is the leveraging of network effects via -- often social -- link propagation along with the ability to make all URI addressed information potentially crawlable, thereby making it transparent via search. The possibility of letting people find your service via an intranet or Web search engine because of the great content it has might seem a little odd at first but then again, that's what makes things work so well on the Web. You can learn about URIs on Wikipedia, and they can be a SOA's best friend.

Using Ajax as the face of your SOA. This point emphasizes yet more service-consumption in the browser. Why? Because the browser model, with our newest high-speed corporate networks, fast desktops, and latest browsers, has finally becoming a very capable way of distributing software and associated updates. No admin rights are required to run an Ajax application in the browser, no plug-ins must be distributed, and users find the experiences they offer compelling. And the Ajax technique requires Web services into order function because an Ajax app is nothing more than a Web page brought to life by Javascript. And if the page upon which the Ajax application resides ever reloads, the application is then forced to quit immediately. Because of this, Ajax software has a major dependence on quality Web services while running, Ajax and SOAs can go hand in hand and potentially complement each other very well.

Finally, I'll point out, as I did in item #1, one key barrier to this unified vision of browser front-end and services back-end is that many SOA services today are just not Ajax consumable. Worse, virtually none are easily consumable by emerging Flash platforms for RIAs such as Flex or OpenLaszlo without a lot of work (or cost), because these platforms have very limited XML processing capabilities such as poor namespace support. This highlights a growing need to sort out the tolerance continuums that are probably too shallow in many enterprises. Note: It's still not easy to develop Ajax apps yet, and so it's worth reading my Seven Things Every Software Project Needs to Know About Ajax for more info about the challenges of applying this fast growing Web services-powered browser application model.

Monetizing Your SOA. On the SOA projects I've been on, many of those who own the systems being opened up as services don't like the results in the short term: more customer service, additional bandwidth and hardware to support unpredictable external use, more testing, and so on. Figuring out ways to meter usage, institute chargebacks, and even charging outright fees to external trading partners and customers allows the necessary negative feedback to discourage irresponsible or profligate use of services. This works well on the Web and the most successful APIs online are metered in some way.

Enable users as service consumers. This also cuts across some of the items above but is best exemplified by the software mashup phenomenon, which describes a method of quickly combining two or more sources of content into a new high-value application. For just the same reason that we have a PC on everyone's desk at work is the reason why almost everyone should be consuming the services in your SOA. Only most users don't have access to the Microsoft Office equivalent of a tool that allows them to mashup or wire together the services you have been producing over the years. The good news is that products specifically geared at enabling the consumption of SOAs by end-users are emerging including IBM's QEDWiki, JackBe's Presto, and many others. In a few years, it's likely that end-users will be one of the largest direct consumers of your services, particularly via syndication. Enable it and encourage it; it's just another way to make your SOA invaluable to the business and generally popular as well.

Virtualization, fast scaling, and on-demand architectures. All of the things driving down the economics of software hosting will allow your SOA to scale up to the Web. Many enterprises view their SOAs and enterprise systems as big, but not compared to the scale of the Web, particularly if provisioning is unmediated (thousands of informal users of your APIs.) Fast adoption is one of the worst nightmares for an SOA that is not well capacity planned and scaled. Just like operations has become a core competency of SaaS and Web 2.0 sites, so too is it in the highly spiky usage model of on-demand services where a successful network effect can cripple your availability and response times.

Offering an SOA as visual services via widgets. The rise of widgets on the Web, making it easy for anyone to put a piece of functionality on their Web page, was a big item in 2006. Widgets also have access to back-end infrastructure (i.e. an SOA) and are snippets of Javascript or Flash badges that allow little bits of data-driven functionality such as stock tickers, corporate news, and other information to reside in any Web page and be fed by back-end services. This is another effective way to put a "face" on an SOA and get it used in many different completely unexpected ways. And because just about anyone can drop a widget onto a Web page, this further unleashes all users -- instead of a few IT staffers -- to help deploy the functionality and data held within an SOA to the far reaches of an enterprise. Additional note: If you're not sure this will be important, check out the hundreds of services available as widgets already on WidgetBox.

Considering JSON as a service option. XML is NOT very fast as I've written about here before, particularly if there is lots of numeric information in the network payload. JSON, the Javascript Object Notation, has risen through the ranks quickly in the last year as a highly compact way to send information on the wire to a Web application. Even the co-inventor of XML, the venerable Tim Bray, has acknowledged the many valid use cases of JSON in networked applications. JSON may not be for you as an architect but your Rich Internet Application developers may very well feel the desire to place pins in your effigy as they try to figure out how to get your sophisticated XML payloads to parse quickly enough -- or in many cases -- to parse at all using the emerging Flash platforms. JSON is fast, compact, and is supremely easy to consume in the browser via Javascript's eval() function.

Encouraging and discovering emergent solutions. Like many are discovering out on the Web, being directly connected to your customers is a completely different proposition than shipping software on a CD. Many SOA practitioners are well aware of this of course, but even the most battle-hardened SOA practitioner would have to go aways to be aware of how extreme it's become online. I've come to describe this tight process of co-evolution via realtime feedback, harnessing user contributions, and becoming a platform that others actually build upon something known as Product Development 2.0. It's not hard to see what happens when users are tightly coupled into the systems that they use and begin using this connection to shape their needs. Even if your corporate SOA doesn't work this way today, it can be made to fairly easily with online metrics and monitoring, though like many SOA issues, governance and control soon become significant issues. Just remember, sites like Flickr deploy changes to production every 30 minutes while monitoring usage and making more changes in almost continuous real-time feedback loops. Other sites are literally letting their users shape the services available from the application itself such as Google with its Web Gadgets offering.

Going further, the concept of Enterprise 2.0 is the front line where much of this particular change will begin taking place in the enterprise, with freeform, social, emergent tools like blogs and wikis that are so general purpose they can used in an almost limitless number of ways. Make no mistake; emergent application platforms are not an edge case trend and are already taking place in your organization with things like the guerrilla deployment of wikis that I'm increasingly seeing in the field. Understanding how an SOA fits into all this (as IBM has, which has labeled their new end-user mashup tool QEDWiki, "the Face of SOA") will be essential for fully leveraging a service-oriented architecture in this environment.

Leveraging the Global SOA. More and more I'm coming across impressive applications that marry the datasets contained within enterprises with the incredibly rich landscape of information out on the Web. And they are primarily impressive because of the data brought in from the Web. I've espoused the concept of the Global SOA, most notably in a cover story for the SOA Web Services Journal, that describes the Web as the richest set of services currently available to anyone, inside or outside the enterprise. It simply no longer makes sense to have an SOA that does not have access to the Global SOA on the Web where hundreds of high-value APIs are available and millions of lesser ones in the form of RSS and ATOM. Infoworld's David Linthicum had some good comments about the convergence of Web 2.0 and the Global SOA, and here is my own exposition with a good diagram that shows the overlap. The challenges around the governance issues of figuring out how to bring in external services safely and provisioning them for use as part of your enterprise SOA. Those who do this successfully can potentially garner an even great uptake on SOA usage as the number of high-value services available internally ramps up quickly.

Most of these items highlight a big trend this year: consumerization of the enterprise as the most effective ideas of the Web 2.0 era begin to flow into enterprises after being proven in laboratory of the Web. There's lot of ideas here, please share your own on what you think about the directions that SOA practitioners will actually take.

A Search for REST Frameworks for Exploring WOA Patterns -- And Current Speaking Schedule

Dion Hinchcliffe — Sun, 10 Sep 2006 12:45:00 GMT

The common patterns for assembling large systems out of smaller systems have been well documented in recent years, with one of the very best examples being Gregor Hohpe's Enterprise Integration Patterns. But as Web sites grow in size and scale -- my favorite quote here being Dare Obasanjo's "My Website is Bigger Than Your Enterprise" -- we're increasingly seeing a push both in the direction of "Web-ness" as well as a preference for radical simplicity in messaging patterns (REST instead of SOAP/WS-*). That doesn't necessarily mean that the grids, server farms, and new 384 processor core "superservers" are neccessarily getting any easier to take advantage of or manage. But we are finally learning how to build systems that are broadly consumable by the limitless variety of the Web's populace, as well as scale on-demand and perform better across the board.

Service-oriented architecture has become the end-game for designing and integrating systems-of-systems within the enterprise. This model as most people know it usually involves SOAP with one or more of the many, many WS-* protocols added to it. However, as we look at bigger and more complex integration scenarios on the Web, this model doesn't seem to be quite as popular or effective. The advent of Rich Internet Applications (RIAs) such as Ajax, Flex, OpenLaszlo, Avalon, and so on, doesn't favor it too much either, since REST tends to be the most consumable Web service type in RIAs by a good margin. And the browser, the preferred application delivery model these days, doesn't speak SOAP at all without serious help, never mind WS-*. I've often quoted the statistic that 80% of all software development should be based on SOA by 2008. If true, this means we clearly have important trends in opposition. What is a software architect to do?

The issue seems to be one that has plagued us in the software business for decades; namely a seemingly incessant urge to predefine all of our solutions before we're even sure what the problem is. The principles of agile software development tell us that we won't understand the problem until we're done developing the software, and that to attempt to design too much up front will result in software (and by extension, standards) that have the wrong features. What this means for the heavyweight standards business is left for the reader, but I'll quote here Groove's George Moromisato when he says that "the chance of any standard being adopted is inversely propotion to its complexity. " Consequently SOAP, and some of the other Web service protocols developed early on in the history the Web, seems to suffer from this very problem.

Enter the growing trend of Web-Oriented Architecture (WOA), an approach to Web services architecture that is, if anything, underprescribed, loose, and therefore very situationally adaptable. WOA has a tolerant, resilient, but steel-reinforced core based on the fundamental protocols of the Web (HTTP, XML, and URI.) And the big advantage of all this is the following: almost certainly the highest levels of interoperability, scalability, and consumability. The latter refers to the fact that REST-based architecture can be used in many more places and many in more ways than traditional SOAs can.

The issue with REST and WOA however continues to be a lack of good description for the lay programmer. Advocates of traditional SOA can point to many well-defined standards documents, descriptions, and reference models. REST and WOA don't have many of those things, each being more of a "style" instead of a standard. Yet, REST and WOA don't go farther than their underlying standards allow and have a malleability and range of motion that over-specified approaches can't possibly match. Unfortunately, Roy Fieldings classic description of REST in Chapter 5 of his dissertion remains impenetrable to the vast majority of developers, though I'm personally working on a couple of projects that might help resolve that.

A quest for REST frameworks to explore WOA

While Nick Gall excellently brought us the vision and concept of WOA, I'm currently focusing on articulating the concept of WOA client patterns, or the best ways to build well-behaved clients in Web-Oriented Architectures. I talk frequently to folks building large systems out on the Web and I've been noticing recurring problems and solutions when they build well-defined, resilient clients that work effectively in the chaotic, unpredictable environment of the Web. From this, I've tried carefully to collect some of the core patterns together and place them into a handful of high level buckets which I've coined WOA/Client and published recently in The Architecture Journal.
Since then, I've slowly begun describing the interior of each pattern. As I do this, I've been attempting to build capable RESTful services that I can create simple, useful reference models for others to see these patterns in action. Surprisingly, I'm still finding that while tooling for REST continues to improve, there still isn't as much out there as I'd like to see. And though fortunately, REST generally doesn't require much to implement, I'm especially looking for that Ruby on Rails-style of simplicity for connecting the service directly to a data-driven source of information. And as you will see below, that's fortunately where we're starting to see some particularly good news.

Some REST frameworks and approaches
While there are dozens of well-known methods for connecting distributed systems together, protocols based on HTTP will be the ones that stand the test of time. And since HTTP is the fundamental protocol of the Web, those protocols most closely aligned with its essential nature will likely be the most successful. So WOA describes architectures based on HTTP, and WOA/Client is the best practices on the client side for participating in a WOA (disclaimer: this latter part is the result of my personal research and is nothing 'official.') In that vein, I've been building REST services to host on this site to demonstrate the power and efficacy of simple, contract-based Web clients that pull data from multiple sources, weave them together into useful forms, and then distribute updates and changes back to their source services. This will be a very common model of service consumption in the near future, particulary in the browser, and the world of mashups is the most clear example of this in the 'wild.'

While I host this site on .NET primarily because SOAP based services are so easy to implement in .NET, it's clear that the open source and Java communities have considerably richer support for REST today. The most advanced framework for REST that I've been able to find is Jerome Louvel's deep and powerful Java-based Restlet framework, available under the CDDL license. It provides full support for REST-based resources, representations, connectors, filters, and routers and also offers excellent examples of REST architectures in the well-written and detailed tutorial.

At the other end of the spectrum is REST-art, a simple and admittedly early-release of a basic, straightforward REST-enabling framework. Currently listed at the top of Sourceforge's project search for REST, REST-art was created by Thibault Ducray, who says "REST-art is a Java library (jar) based on the REST principles (services uses URIs, XML data, http verbs GET, POST, PUT, DELETE). It gives the ability to develop Web Services without using huge (and sometime commercial) SOAP frameworks."

Another sophisticated RESTful framework is the SourceForge project Cognitive Web, which has seemingly advanced capabilities that go well beyond the basics of REST and WOA. The home page of the Cognitive Web says that the project is a, "framework for building, deploying and managing well-described REST-ful Web services, including REST-ful Web Services realizations for RSS, XML Topic Maps, Structured Arguments, and Workflow."

But one of the most interesting REST frameworks I've seen is sqlREST, an easy-to-use framework that allows a set of relational database tables to be exposed as REST-compliant Web services. While not a production-ready release, it's the concept of sqlREST that is the most interesting; an automatic mapping of database information in relational format to an interactive view of the same data in XML/HTTP format, REST-style. As data-driven Web applications become more and more common, having an easy way to turn your Web application into an open platform will be a useful (and essential) technique for growth. sqlREST makes this possible with little development effort.

Finally, Matt Biddulph did a terrific job last November writing up how Ruby on Rails can be used to quickly and effectively build REST services with Rails using the built-in REXML facility. Ruby on Rails is famously known for reducing abstraction impedance across the board, and Matt's tutorial reminds us again how useful Rails is for making routine, yet normally complicated, programming tasks easy, at least once you learn the Rails way.

Speaking Schedule on Web 2.0, SOA, and WOA

I cover all this material, and a lot more, in my talks on the next generation of Web, IT, and business. While a few of my presentations are very technical and will cover the intricacies of SOA/Web 2.0 co-evolution or how to build scalable Ajax applications using WOA, I also talk about how the latest trends and service models on the Web are changing the way we work, live, and otherwise interact with each other. If you're going to any of these events, please be sure to stop me and introduce yourself.

"21st Century Professional and Civil Society" Conference - Dallas, Texas - Saturday, September 16th, 2006 - I'll be speaking on the role of Web 2.0 trends and technologies and how they will affect professional and civic life in the 21st century. As well as covering the basic of Web 2.0 and moderating a panel, I'll be covering the innovative application of Web 2.0 platforms such as blogs and wikis to document important events like the genocide in Darfur to the great story of the use of emergent collaboration on the Web to locate one hundred thousand Hurricane Katrina survivors in a short few days.

Interop New York 2006 - New York City, New York - Tuesday, September 19th, 2006 - I'll be giving one of my usual Web 2.0 in the enterprise presentations, which are invariably pretty popular. The session description says: "From a buzzword last year, to the thematic rallying cry of 700+ startups this year, and now apparently heading for mainstream adoption, Web 2.0 is the topic du jour for the consumers and businesses both. This session attempts to demystify the trend with practical description and examples along with a succinct explanation of Web 2.0. The second half explores how Web 2.0 is being used by companies in various ways to improve internal productivity and innovation, not to mention how to involve customers and reinvent the concept of customer service and marketing."

The New New Internet - Tyson's Corner, Virginia - Tuesday, September 20th, 2006 - I'll be moderating some panels for this Web 2.0 conference which I'm helping organize in the Washington, DC area. We have Michael Arrington's TechCrunch, Harvard Business School's Andrew McAfee (of Enterprise 2.0 fame), Michael Platt from Microsoft's Architecture Strategy Group and a bunch of other great speakers. I do hope to see you there, it'll be the first major Web 2.0 in the enterprise conference on the East Coast and at the tony Ritz-Carton right in middle of the Silicon Beltway.
CIPS (Canadian Information Processing Society) Winnipeg ‘06/’07 Kick-Off Dinner - Winnipeg, Canada - Wednesday, September 21st, 2006 - I'll be the evening's speaker and the subject will be Social Computing, a topic about how the widespread use of Web 2.0 technologies tends to create long term changes in the societies that apply them, particularly a shift from push to pull-based models for management and self-organization. Fascinating stuff. This is a topic I've covered quite a bit in recent months including most notably on ZDNet, where it was Slashdotted, and at the SOA Web Services Journal.

AjaxWorld University Bootcamp - Santa Clara, California - Monday, October 2nd, 2006 - I'll be leading up an all-day, high-intensity bootcamp on how to build "addictive" Ajax applications using the latest frameworks and design patterns. We'll cover Dojo, Prototype, Script.aculo.us, as well as learn how to design irresistible user interfaces with concepts inspired by Kathy Sierra's Addictive User Experiences material. It will be the fastest way to learn Ajax best practices you can find anywhere. I hope to see you there.

AjaxWorld Conference and Expo - Santa Clara, California - Tue.-Wed, October 3rd-4th, 2006 - Not only will the top 70 or so folks in Ajax and RIAs be there but a mini-conference on Ruby on Rails and Flex will be on site too. The list of well-known folks speaking is amazing and combined with the bootcamp above, should be the biggest Ajax event of the year. Disclaimer: I'm the conference chair of AjaxWorld this year. I'll also be speaking at a session titled Leveraging the Web 2.0 Movement. Description: "The emerging Web 2.0 movement is emphasizing the same things as SOA -- which is rapidly emerging as the best practice for building services, reusing IT assets, and providing open access to data and functionality. Web 2.0 and SOA are in fact beginning to converge. Web 2.0 describes the Internet as a "Global SOA" while at the same time laying out practices for building lighter weight SOAs based on REST and RSS. Furthermore, Web 2.0 and online services are expected to become the dominant application model with behemoths like Microsoft and Google fighting for dominance. More interestingly, this phenomenon is actually happening now and helping businesses deliver more value to their customers every day. This fact-filled session will discuss the latest trends citing industry sources, case studies, and other well-known examples."

Office 2.0 Conference - San Francisco, California - Tuesday, October 12th, 2006 - Ismael Ghalimi has organized a terrific conference with a leading cast of speakers on the topic of the next generation of Web-based business software. Ismael calls this concept Office 2.0 and it's rife with important SaaS, Ajax, and Web-based implications for the software business. I'll be moderating a panel on Tuesday morning titled Making Office 2.0 Enterprise Ready.

That's it for now, expect some more WOA/Client patterns soon, particularly a rigorous exploration of contract checking with a sample REST service, instead of the SOAP one we did last time. I'll also be updating my publishing schedule soon; we have some exciting material on SOA/Web 2.0 converence as well as REST coming in a couple of months as well.