Service Type Specifications

Over the last weeks I've been busy with finding a suitable approach towards REST Web services descriptions. Not quite there yet, but close! Here is my list of essential aspects a service type specification must address:

1. Overall purpose of the service (aka the realized business process role)


2. What client goals does the service enable (aka protocol operations)


3. When can the client expect certain goals to be available (aka partial ordering of application states)


4. What does the client have to do to 'execute' a certain goal (aka hypermedia semantics)

Separation of Concerns and Replication

In an integration scenario that involves the replication of items from one system (the master) to another (the slave) the direction of the communication is an interesting issue.

First, there is the question, whether the initial creation of a replicated item in the slave should be performed by a request from the master to the slave or by a request from the slave to the master. The former will often be chosen when there is a business rule involved that should trigger the replication (e.g. replicate all employees in the slave system whose contract ends in 6 months). The latter is more likely to be used in a scenario where the slave replicates all new items in the master, in other words, where the slave keeps the overall set of items in sync.

Second there is the question how updates of the items in the master are made available in the slave. This is the classic distinction between polling and publish/subscribe: Should the master notify the client of updates or should the slave poll for updates?

In addition, there is sometimes a desire to notify the master about changes the slave makes to the replicated items. The intention of this scenario is not the bidirectional synchronization of master and slave but reflects the master's interest in any changes the slave makes.

What is the guiding principle to make an informed design decision regarding the direction of communication in such a replication scenario? The answer is separation of concerns with the goal of simplicity and avoiding unnecessary coupling. This leads to the question which of the systems should for which communication play the server role and which one should play the client role?

Every communication is initiated to achieve some goal and the correct separation of concerns puts the component that acts to achieve the goal into the client role and the other component into the server role.

Applied to the above scenarios this means that for initial replication and subsequent updates the slave system should act as a client and the master should act as a server. All the necessary resources to be provided by the sever (e.g. providing list of all items that match a business rule) can be derived.

Applied to the scenario where the master is interested in updates from the slave it means that the master should act as the client because it is the master that has the goal (the slave does not care at all). This leads to certain resources that need to be exposed by the slave.

I have seen at least three production systems that had this issue and all of them did it wrong and would have greatly benefited in terms of simplicity and loose coupling from following the above principled design path.

Related is Roy's post Paper tigers and hidden dragons.

History of Fragment Identifiers

Everything you ever wanted to know about fragment identifiers - an in-depth coverage by Roy back in 2002.

300 and 406 Response Bodies

Reading through some older postings, I came across this by Roy:

That is why we have a WebDAV working group.  Both the 300 and 406 response bodies were left unspecified because the intention was that they be specified by a group that actually had time to study the problem in detail and come up with a [hopefully] better solution than some off-the-cuff invention of mine.  It was one of the WebDAV to-do items, last time I checked.

Has this ever been done? Maybe it is time to create some standard 300 and 406 response media type?

UPDATE: Umm, should have searched a bit more I guess: RFC2295. What is left is to tell clients to look for Alternates headers in 300 and 406 responses.

ESB Decoupling Capabilities and the Elegance of REST

In his presentation "The Role of the ESB" Mark Richards makes the point that the most significant capability of an ESB is that it enables the separation of "Business Services" from "Service Implementations" (at 00:11:25). "Business Services", he says, "are the ones that make sense to the business" while "the implementation service [...] is the one we actually code". As an example he contrasts "PlaceTrade" and "saveTradeOrder()" and talks about how PlaceTrade is much more stable than saveTradeOrder() because the former is a business semantic and not an implementation detail.

While it is truly astonishing what amount of tooling and complexity the slide conveys in order to provide a business service 'PlaceTrade' it is something else that struck me as being really interesting:

It is obvious that, in order to provide a 'PlaceTrade' service, there needs to be implementation details that should not be conveyed to the user of 'PlaceTrade'. (According to Richards, SOA wonderfully solves that problem by using an ESB that translates the business service into an implementation service call. Can't help it...enterprisey comes to mind...)

REST takes a different approach: in a RESTful system, the implementation details that correspond to the translation of 'PlaceTrade' to 'saveTradeOrder()' manifest themselves in the form of the resource model of a service (the actual resources the service chooses to make available). This is a direct consequence of realizing business functionality through a uniform interface.

Interestingly, in a RESTful system, clients are inherently decoupled from the services' resource models - the contract between client and server is expressed solely through hypermedia specifications, independent of any actual service implementation.

In other words: SOA makes use of ESBs to achieve a decoupling that is already inherent in REST per design.

POST /trades
Content-Type: application/trade-order

<trade-order>
....
</trade-order>

Service Descriptions and Legal Contracts

The intermediate result of the thoughts triggered by the recent interchange on the SOA list are here. Comments welcome!

Formal Contracts

When clients in a RESTful system are not ultimately driven by a human then the expectations they make about the resource model of the services and the available representation formats will inevitably be manifested in the client side code to some extend.

While the client developer can go along way in limiting the assumptions made about the server at some point the client code just needs to explicitly assume that there is for example a 'place order' state traversal (aka 'link' or 'form') when interacting with a service were you can order stuff. The client can plan for various ways in which the server could express the link or form but ultimately the client code will contain some form of orderProcessorUri = clientLib.lookupUri(...expression to find order processor URI...).

If the traversal option cannot be found the client will break without a possibility to automatically react. This is where a human user would change the online store or call the hotline.

Steve Jones is concerned about the development process related- and legal implications of the lack of a formal means for documenting such implicit contracts created between client and server once the client manifests its assumptions in its own code or configuration. I think he is making some valid (and too often ignored) points.

I have been working on related questions during the past weeks and currently see the following issues:

• While the freedom that is usually (deliberately) given to servers in Internet specifications improves evolvability it is not a sufficient basis for establishing legal contracts between service consumers and service providers. If you make a serious investment into implementing a client to some service then RFC 2119's SHOULD is probably not the best contractual basis.
• If a service makes use of a variety of media types, extensions, link relations, categories, etc. then how would one express this in a formal way?
• If a client is dependent on the presence of some Atom extension in Atom entry documents for example, how could the service provider state that they are committed to providing that extension?

It is important though to emphasize the general goal of maintaining the flexibility induced by REST's constraints and at the same time to make explicit and to formalize the inevitable coupling that occurs when non-human clients interact with services.

This is important not only for development-process and legal matters but also for IT governance and Service Management issues and critical for the adoption of REST inside the enterprise.

Hidden Contracts and Governance

In a RESTful system the contract between client and server is expressed solely in the form of hypermedia specifications[1]. On the Web this works fine, because there is usually always a human being involved that can act as the ultimate arbiter of semantics when things need to be decided that happen not to be covered by the involved hypermedia specifications.

Examples:

1. When I want to buy a book, what hypermedia is telling me that www.amazon.de is a place to go?

2. When the Amazon site is not reachable, how do I find out where else I could buy a book?

3.How do I know what 'buying a book' means and what the basic sequence of operations is that need to be executed in order to achieve the overall goal?

4. How do I figure out, what the new appropriate sequence of operations is should Amazon significantly change its understanding of the flow of a book purchase?

All of the above are ultimately solved by a human being in the case of the Web and usually don't cause any trouble.

3. is particularly interesting because the fact that no hypermedia specification even touches that question reveals that there is a very high level, common sense contract in operation between human clients and some Web applications.

4. is also interesting because economic sense will usually keep e.g. a book seller from changing the flow of operations in a way that causes the potential buyer to not being able to buy anymore.

How does all this relate to IT governance?

When one applies REST (HTTP that is in practice) in an enterprise IT context, these hidden contracts matter because they either need to be covered by hypermedia specifications (e.g. for the discovery/equivalence examples 1. and 2.) or at least need to be made explicit so they become manageable in an IT governance sense.

Example 3. relates to the general question of how types of services (e.g. 'procurement seller role' or 'repository') can be expressed to provide guidance to client developers regarding the intended sequence of operations.

Example 4 relates to the question of how change can be managed not of the pertaining hypermedia specifications (this is easy!) but of their composition (which is what effectively forms the service type).

Besides being able to frame the problem and ask the questions I do not yet have definitive answers - comments and thoughts are very welcome.

lass="Apple-style-span" style="font-size:medium;">[1] The term 'hypermedia specification' is meant as a general expression for specifications of media types, media type extensions, link relations and 'little' things like category definitions (used for example by Atom and NewsML2).

(Also posted in the XING REST Group)

Taylor, Medvidović, Dashofy. 2010. Software Architecure

The book just arrived and I think it is one of those worth an in-depth study!

I like the caution note on page 421:

"Numerous Web sites and books purport to characterize or exemplify REST principles. The reader should be very cautious, as many sources misrepresent or mischaracterize REST."

Sadly so..

Product Modeling at W3C

This seems sort of interesting and probably has some potential to be used in Web architecture-based procurement collaborations. Is there any serious industry involvement in this?

I wonder if OWL and friends will eventually obfuscate the real goal; wouldn't a decent media type be a better way to approach this?

Formal Hypermedia Contexts

Formally, a hypermedia context can be understood as a conjunction of predicates that evaluates to true or false for a given URI. The predicates can be expressed as one of the following:

• unary link, e.g service(x)


• binary relationship, e.g. edit-media(u1,x)


• pattern match on representation of x, e.g. r-match( xpathExpr , x)


• pattern match on occurrance of x in representation, e.g match( xpathExpr , x)

Logical sentences build from such predicates can be evaluated against the current state of the client to either test if a given URI matches the denoted context or to find all URIs known to the client that match the predicate conjunction.

The next step will be to express the operations that are currently stated in the form of prose in hypermedia specifications in terms of the above predicate conjunctions and HTTP operations. For example could AtomPub's 'create entry' operation be expressed as the combination of the predicate conjunction match( '/service/workspace/collection@href',x) and POST. Where the match predicate evaluates to true for any URI x that is found as the value of the XPath expression.

This needs more precision of course, but it's a start.

RFC 5545

iCalendar is now an IETF Proposed Standards Protocol (superseedes RFC 2445). Does anyone know whether this is just fixing things or whether there are additional semantics that increase the expressiveness?

Hypermedia Context

Descriptions of REST interfaces often use the notion of types when talking about the exposed resources. The Atom Publishing Protocol for example includes a section that provides a resource classification. Such classifications are not against the RESTfulness of an interface as long as the classes are expressed in terms of the knowledge a client has acquired about a resource. The Atom Publishing Protocol for example states that "a Resource whose IRI is listed in a Collection is called a Member Resource". This is a valid classification from the point of view of its RESTfulness.

In order to emphasize that such classification can only be done in terms of the acquired application state of the client I suggest the term "hypermedia context" instead of "classification". REST interface descriptions could then for example state: "The hypermedia context of a resource is called Member if its URI is listed in a collection". There is probably also value in distinguishing the client-acquired knowledge about a resource from the context used in the specification by calling the former acquired hypermedia context or just hypermedia context and the latter named hypermedia context.

I can also see possibilities to formalize hypermedia contexts along the lines of predicates, for example as an AND combination of known links, appearance of links in received hypermedia and (XPath-)matches on the representations of resources.

But I'll address that in a follow up.

Location Header Semantics

Depending on the response's HTTP status the Location header has slightly different impact on the client's application state. The following table summarizes what I extracted from the HTTP spec.

Status	Semantic	Comment
201 Created	URI of created resource	Client is notified that a new resource has been created an can dereference the URI to see what has been created, e.g. to verify its expectations. ETag can be used to provide current ETag of created resource
204 No Content	URI of created resource(?)	Clarification pending
300 Multiple Choices	URI of server's preferred variant
301 Moved Permanently	New permanent URI
302 Found	Temporay URI to use for requests to the intended resource
303 See Other	Response to the POST request is found at another URI	If client follows redirect and if the client already knows the target resource the 303 works as a notification that the previous POST has changed the target resource.
305 Use Proxy	URI of proxy to use for the request.
307 Moved Temporarily	Temporary URI to use for requests to this resource.

201 and 303 provide the grounds for interesting generic client behavior. About the nature of 305 I am not sure yet but the server's ability to redirect a client to an intermediary smells interesting.

REST Interface Implementation Design

It occurred to me today that there is a subtle design artifact that sits in between the specification of a REST API and its actual implementation. Let me try to explain what I mean:

The specification of a REST API consists of the media types, extensions and link semantics that a potential client must understand to be able to meaningfully interact with the API. The combination of Atom and the Atom Publishing Protocol is a good example of this.

The implementation of a REST API maps the 'back end layer' to a number of resources that provide access to the 'back end layer' in terms of the semantics of the REST API.

This looks like the expected differentiation between interface and implementation, much like you implement an OO interface with a class.

Interestingly, in the case of REST there is a third design artifact that resides somewhere in the middle between interface and implementation: the design decision of what resource are actually exposed and which of the interface semantics are actually used in which circumstances. It could be argued that this decision is part of the implementation, but by distinguishing the two it becomes clearer that there are many potential ways how different systems can manifest a given REST interface.

A class that implements an OO interface will always just implement the methods that constitute the interface, no decisions besides the actual implementation, no surprises. When implementing an Atom Publishing Protocol service, there is an unlimited number of ways to organize feeds and add links to feed or entry documents. One can even build two entirely different APP facades around a given back end but still use the same code to map back end resources (general sense) to feed and entry documents.

When REST API specifications define what resources a client will find on the server side and what type they have then this is an indication that the specification (wrongly) includes the server-side design artifact described above. Such specification elements must be removed from the API to make it RESTful.

Parameters are Link Relation Specific

I used to view REST APIs to be constituted by media types, extensions, link relations and parameters (query-, form- and URI template parameters). Reading through Roy's checklist again today I realized that he suggests a superior approach: By making the parameters link relation specific the problem that parameters are usually not bound to a name space goes away and there is one less class of design artifacts.