I just got back from a very nice long vacation on the island of Crete. It was amazing how peaceful, quiet, and restful a Mediterranean island can be. And there was hardly an Internet cafe to be found. As the joke there goes, the 'net is something they catch fish with in Crete.
In any case, now that I'm back I plan to jump back into the Service Description Language (SDL) comparison, probably by starting on the REST side of things, but perhaps by starting on the comparison work for SSDL, the SOAP Service Description Language. I also plan on adding the recently emerged WAIF to the SDL mix, taking the total number of description languages we'll be looking at to 15. Undoubtedly we'll learn many things along the way, other than Ruby is so far the coolest language in which to write loosely coupled web services.
In any case, on my return, imagine my surprise at finding a recently raging discussion about the relatively poor performance of SOAP web services vs. real distributed computing solutions. And from folks that have some genuine expertise in the matter too. At times, I do find it hard to believe this hoary old debate is still around, but unfortunately there are lots of reasons why it is, which I'll get into in a moment. Admittedly though, I do find one of the points against SOAP fairly compelling from an earned value perspective...
Figure 1: Objective numbers in the distributed computing performance debate are elusive
The argument is this: Given an application that would consume $5K worth of Internet bandwidth using highly efficient means like CORBA or ICE, the same bandwidth with WS-I web services costs upwards of $100K if SOAP is 20 times less efficient than the more optimized solution. And SOAP might even be less efficient than that.
This is an argument that reasonable people can find a lot of validity in. How can you build a service-oriented architecture to meet a constellation of internal and external stakeholder requirements if you have incredibly below average (or very expensive) bandwidth usage? The answers are quite interesting in the end. Since actual real-world performance and associated large hidden costs of poor performance have a huge impact to any SOA effort, it's worth taking a quick foray into the subject to see if we can uncover some ground truth and contribute clarity.
The seemingly incredible pokiness of your average SOAP stack is almost legendary. Check this paper out, as well as this one for a small but representative sampling of SOAP getting a total pasting when compared to CORBA or RMI. There are a bunch of factors that architects must take into consideration to have the full performance equation. Since acceptable performance that is sustained over time, including likely adverse situations, is generally what most architects are measuring for, let's take a look at the likely ingredients that affect distributing computing performance.
1) Payload Efficiency. What is the relative size of the communication payloads between nodes on the network and how efficiently is it represented?
2) Protocol Stack Efficiency. What bandwith overhead above and beyond carrying the payload is involved. How much metadata is transferred, cost of compression, decompression, encyprtion/description, number of round trips required, data conversion of numbers, etc.
3) Avg. Available Bandwidth. Not the raw maximum, but given that you're sharing a network, how much is generally available to transfer payload?
4) Distance Between Hosts. This can really matter as the number of network hops and geographic distance grows.
5) Tuning: Did you tune your implementation or did you provide it to your customers to use without profiling the result?Small details in your implementation can add huge performance penalties especially over a large number of requests.
These represent the nominal factors that will affect your distributed computing performance and you can measure these with some effort and time. Ideally, in a mature professional field there would be a well established body of knowledge on the relative performance of existing techniques, styles, standards, and even products from which to start from. Unfortunately, staggeringly poor information is actually the norm instead of useful information. Diligently searching Google will find you very little in the way of hard numbers, except between well known open source solutions.
Interestingly, this means one of the bigger obstacles to understanding the performance of web services solutions are the vendors themselves. Virtually all software licenses prevent their holders from publishing unauthorized benchmarks publically. This means it's difficult to compare apples to apples (SOAP vendors) much less apples to oranges (SOAP vs. CORBA, etc) unless you do it all yourself. While this restriction on information is to protect the vendor from having invalid or slanted benchmarks from being published, it stops any information about the current state-of-the-art in web service performance from being generally known. Outside of the aforementioned open source toolkits that is. Is this just another instance of open source being superior to commercial software? Quite possibly.
In the meantime, the web services industry may just end up where the database industry did, by creating a widely recognized benchmarching organization. This gave us the mostly impartial Transaction Processing Council and their now legendary (and still up-to-date) TPC-C benchmark list of the fastest database solutions available. TPC-C isn't perfect but has always been a useful repository of the fastest database hardware and software products.
Note: The TPC has recently inaugurated a TPC-App benchmark which requires all data exchange to be in SOAP and XML but sadly, there is only one entry currently.
Another case in point is that Microsoft's new WS-* product (the stack formerly known as Indigo) has aggressive web service performance features either standardized on MTOM/XOP or using their proprietary "super fast" format. Microsoft may have solved SOAP performance concerns once and for all, but it's impossible to tell without any way to measure it except to do it yourself and compare it against the others. There isn't even any workable way for them to crow about how much faster it is than other approaches and products. Perhaps even worse, vendors can't optimize or compete on benchmarks that aren't widely known, leaving us with web service performance as a grey area unlikely to recieve much attention. A sad state of affairs indeed.
Where does this leave us? Not with many choices. Unless benchmarks like TPC-App get buy-in from the industry and organizations begin to submit their results with a variety of solutions, hard performance data will remain an unknown soldier. It's also obvious to everyone that SOAP is simply nowhere near the fastest messaging protocol in the world, and it's not meant to be. It's designed to be the most interoperable and highly extensible, and that is an admirable and extremely useful goal for many situations. It's just like the reason that we no longer program in assembly language: It may result in the fastest programs possible but there are so many more desirable goals such as maintainability, speed of development, and lower defect rates that just plain exclude it from consideration. Yes of course.... large-scale SOA architectures with mission critical elements will tend to have performance high up on the list of prioritized requirements. And that's when having poor information about your choices throws up costly and time consuming barriers to your success. As web services practitioners, we must find a way to fix this missing view of our landscape as soon as possible
Technorati: Enterprise Architecture, Computers and Internet, Programming, Web Services