Moved Blogs

I have moved blogs to my company's server, please continue following over there:

http://www.nordsc.com/blog/

Thanks.

Service Type Specifications II

Wow, this question was much harder to get straight than I expected:

What constitutes the specification of a service type?

First things first, though! In order to provide a principled answer to that question, another one needs to be answered before:

What is the purpose of a service type specification?

Purpose of Service Type Specifications

1. Service type specifications provide the information client developers need to implement a client for services of the described type. If you hand the service type specification to a developer she should be able to know exactly what to do and what to reasonably expect from any instance of that service type. There should be no further knowledge required (except for following any included references, for example, to media type specifications, of course).

2. Service type specifications provide the information necessary for implementing instances of the specified service type. There should not be any further information required except for implementation specific details behind the service boundary of course.

3. Service type specifications provide the information service owners and maintainers need in order to understand in which way the server can evolve without breaking clients. This is redundant with 2. but mentioning it explicitly emphasizes where exactly the contract is established between client and server owners. Anything that is not specified in the service type specification or referenced material is not part of the contract and constitutes no obligation by either party.

4. Service type specifications enable service discovery by type. Clients that wish to interact with a certain kind of service can use the information provided by the service type specification to detect when they see a service that is of the desired type. The necessary information should be available as a response to the published service URI either by analyzing the set of initial transitions (goals) provided by the service or by looking at the service document's media type (see below).

Having laid out the purposes of a service type specification, we can now make principled decisions about what should be part of a service type specification.

Elements of Service Type Specifications

Here is the short answer (rationale follows below):

Service Type Specifications define the set of hypermedia specifications (media types, link relations, etc.)  used by the service and information about the minimal initial set of available transitions (goals).

The latter can optionally be expressed as a media type, too (a service type specific service document type), which simplifies the definition to 'a set of media types' but leads to the creation of a new media type for a given kind of service.

The essential aspect of the above definition is that the client needs to know what the media types are it needs to understand in order to interact with the service.

So, why is that?

The ideal situation would be to say nothing about the service type at all, just agree on a set of media types that make sense to be understood in general and implement all or any of them in the clients as desired. The problem with this approach is that it does not address purpose 1. above; a client developer would not have any clue what a service is doing or how to interact with it. There would not be any notion of a service type at all; just individual hypermedia semantics (media types, link relations etc.).

But even if the client developer was provided with some means of a service type description (in the form of a dedicated service media type or a set of initial transitions) - see purpose 4. above - there would still be no way for the client developer to have any clue what can be done with the service beyond the initial transitions. Knowing the set of media types provides that clue.

The issue of guiding the service developer (purpose 2.) is addressed because the service type tells the service developer exactly what media types etc. are available to him to solve the given implementation task.

Purpose 3. above is addressed by the fact that it is not possible to remove a hypermedia specification from the specified set without incompatibly changing the semantics of the service type. Service owners are therefore free to evolve the service by adding hypermedia specifications (or extending extensible ones) but may not remove any.

The purpose of discovery (4. above) can be addressed by using a generic service description media type such as Atom Service documents (application/atomsrv+xml) and describing a minimal set of available resources. For example, an ITIL-conforming help desk service type could be specified as providing at least three collections (identified by categories) containing and accepting submissions of Incidents, Problems and Change Requests. Clients (including service registry crawlers) would know they come across an instance of that 'ITIL helpdesk service' when they see an Atom Service with the specified collections.

Alternatively, a new service document media type could be minted (e.g. application/helpdesksrv+xml) to identify the service type. The format of that type would be defined to provide links to the necessary resources. This leads to simpler discovery/registry mechanism but also might cause explosion of media types.

Using a combination of both might actually be best, such as application/atomsrv+xml;profile=helpdesk.

Service Type Specifications

Over the last weeks I've been busy with finding a suitable approach towards REST Web services descriptions. Not quite there yet, but close! Here is my list of essential aspects a service type specification must address:

1. Overall purpose of the service (aka the realized business process role)
2. What client goals does the service enable (aka protocol operations)
3. When can the client expect certain goals to be available (aka partial ordering of application states)
4. What does the client have to do to 'execute' a certain goal (aka hypermedia semantics)

Separation of Concerns and Replication

In an integration scenario that involves the replication of items from one system (the master) to another (the slave) the direction of the communication is an interesting issue.

First, there is the question, whether the initial creation of a replicated item in the slave should be performed by a request from the master to the slave or by a request from the slave to the master. The former will often be chosen when there is a business rule involved that should trigger the replication (e.g. replicate all employees in the slave system whose contract ends in 6 months). The latter is more likely to be used in a scenario where the slave replicates all new items in the master, in other words, where the slave keeps the overall set of items in sync.

Second there is the question how updates of the items in the master are made available in the slave. This is the classic distinction between polling and publish/subscribe: Should the master notify the client of updates or should the slave poll for updates?

In addition, there is sometimes a desire to notify the master about changes the slave makes to the replicated items. The intention of this scenario is not the bidirectional synchronization of master and slave but reflects the master's interest in any changes the slave makes.

What is the guiding principle to make an informed design decision regarding the direction of communication in such a replication scenario? The answer is separation of concerns with the goal of simplicity and avoiding unnecessary coupling. This leads to the question which of the systems should for which communication play the server role and which one should play the client role?

Every communication is initiated to achieve some goal and the correct separation of concerns puts the component that acts to achieve the goal into the client role and the other component into the server role.

Applied to the above scenarios this means that for initial replication and subsequent updates the slave system should act as a client and the master should act as a server. All the necessary resources to be provided by the sever (e.g. providing list of all items that match a business rule) can be derived.

Applied to the scenario where the master is interested in updates from the slave it means that the master should act as the client because it is the master that has the goal (the slave does not care at all). This leads to certain resources that need to be exposed by the slave.

I have seen at least three production systems that had this issue and all of them did it wrong and would have greatly benefited in terms of simplicity and loose coupling from following the above principled design path.

Related is Roy's post Paper tigers and hidden dragons.

History of Fragment Identifiers

Everything you ever wanted to know about fragment identifiers - an in-depth coverage by Roy back in 2002.

Reading through some older postings, I came across this by Roy:

That is why we have a WebDAV working group.  Both the 300 and 406 response bodies were left unspecified because the intention was that they be specified by a group that actually had time to study the problem in detail and come up with a [hopefully] better solution than some off-the-cuff invention of mine.  It was one of the WebDAV to-do items, last time I checked.

Has this ever been done? Maybe it is time to create some standard 300 and 406 response media type?

UPDATE: Umm, should have searched a bit more I guess: RFC2295. What is left is to tell clients to look for Alternates headers in 300 and 406 responses.

ESB Decoupling Capabilities and the Elegance of REST

In his presentation "The Role of the ESB" Mark Richards makes the point that the most significant capability of an ESB is that it enables the separation of "Business Services" from "Service Implementations" (at 00:11:25). "Business Services", he says, "are the ones that make sense to the business" while "the implementation service [...] is the one we actually code". As an example he contrasts "PlaceTrade" and "saveTradeOrder()" and talks about how PlaceTrade is much more stable than saveTradeOrder() because the former is a business semantic and not an implementation detail.

While it is truly astonishing what amount of tooling and complexity the slide conveys in order to provide a business service 'PlaceTrade' it is something else that struck me as being really interesting:

It is obvious that, in order to provide a 'PlaceTrade' service, there needs to be implementation details that should not be conveyed to the user of 'PlaceTrade'. (According to Richards, SOA wonderfully solves that problem by using an ESB that translates the business service into an implementation service call. Can't help it...enterprisey comes to mind...)

REST takes a different approach: in a RESTful system, the implementation details that correspond to the translation of 'PlaceTrade' to 'saveTradeOrder()' manifest themselves in the form of the resource model of a service (the actual resources the service chooses to make available). This is a direct consequence of realizing business functionality through a uniform interface.

Interestingly, in a RESTful system, clients are inherently decoupled from the services' resource models - the contract between client and server is expressed solely through hypermedia specifications, independent of any actual service implementation.

In other words: SOA makes use of ESBs to achieve a decoupling that is already inherent in REST per design.

POST /trades
Content-Type: application/trade-order

<trade-order>
....
</trade-order>

Service Descriptions and Legal Contracts

The intermediate result of the thoughts triggered by the recent interchange on the SOA list are here. Comments welcome!

Formal Contracts

When clients in a RESTful system are not ultimately driven by a human then the expectations they make about the resource model of the services and the available representation formats will inevitably be manifested in the client side code to some extend.

While the client developer can go along way in limiting the assumptions made about the server at some point the client code just needs to explicitly assume that there is for example a 'place order' state traversal (aka 'link' or 'form') when interacting with a service were you can order stuff. The client can plan for various ways in which the server could express the link or form but ultimately the client code will contain some form of orderProcessorUri = clientLib.lookupUri(...expression to find order processor URI...).

If the traversal option cannot be found the client will break without a possibility to automatically react. This is where a human user would change the online store or call the hotline.

Steve Jones is concerned about the development process related- and legal implications of the lack of a formal means for documenting such implicit contracts created between client and server once the client manifests its assumptions in its own code or configuration. I think he is making some valid (and too often ignored) points.

I have been working on related questions during the past weeks and currently see the following issues:

• While the freedom that is usually (deliberately) given to servers in Internet specifications improves evolvability it is not a sufficient basis for establishing legal contracts between service consumers and service providers. If you make a serious investment into implementing a client to some service then RFC 2119's SHOULD is probably not the best contractual basis.
• If a service makes use of a variety of media types, extensions, link relations, categories, etc. then how would one express this in a formal way?
• If a client is dependent on the presence of some Atom extension in Atom entry documents for example, how could the service provider state that they are committed to providing that extension?

It is important though to emphasize the general goal of maintaining the flexibility induced by REST's constraints and at the same time to make explicit and to formalize the inevitable coupling that occurs when non-human clients interact with services.

This is important not only for development-process and legal matters but also for IT governance and Service Management issues and critical for the adoption of REST inside the enterprise.

Hidden Contracts and Governance

In a RESTful system the contract between client and server is expressed solely in the form of hypermedia specifications[1]. On the Web this works fine, because there is usually always a human being involved that can act as the ultimate arbiter of semantics when things need to be decided that happen not to be covered by the involved hypermedia specifications.

Examples:

1. When I want to buy a book, what hypermedia is telling me that www.amazon.de is a place to go?

2. When the Amazon site is not reachable, how do I find out where else I could buy a book?

3.How do I know what 'buying a book' means and what the basic sequence of operations is that need to be executed in order to achieve the overall goal?

4. How do I figure out, what the new appropriate sequence of operations is should Amazon significantly change its understanding of the flow of a book purchase?

All of the above are ultimately solved by a human being in the case of the Web and usually don't cause any trouble.

3. is particularly interesting because the fact that no hypermedia specification even touches that question reveals that there is a very high level, common sense contract in operation between human clients and some Web applications.

4. is also interesting because economic sense will usually keep e.g. a book seller from changing the flow of operations in a way that causes the potential buyer to not being able to buy anymore.

How does all this relate to IT governance?

When one applies REST (HTTP that is in practice) in an enterprise IT context, these hidden contracts matter because they either need to be covered by hypermedia specifications (e.g. for the discovery/equivalence examples 1. and 2.) or at least need to be made explicit so they become manageable in an IT governance sense.

Example 3. relates to the general question of how types of services (e.g. 'procurement seller role' or 'repository') can be expressed to provide guidance to client developers regarding the intended sequence of operations.

Example 4 relates to the question of how change can be managed not of the pertaining hypermedia specifications (this is easy!) but of their composition (which is what effectively forms the service type).

Besides being able to frame the problem and ask the questions I do not yet have definitive answers - comments and thoughts are very welcome.

[1] The term 'hypermedia specification' is meant as a general expression for specifications of media types, media type extensions, link relations and 'little' things like category definitions (used for example by Atom and NewsML2).

(Also posted in the XING REST Group)

Taylor, Medvidović, Dashofy. 2010. Software Architecure

The book just arrived and I think it is one of those worth an in-depth study!

I like the caution note on page 421:

"Numerous Web sites and books purport to characterize or exemplify REST principles. The reader should be very cautious, as many sources misrepresent or mischaracterize REST."

Sadly so..

Product Modeling at W3C

This seems sort of interesting and probably has some potential to be used in Web architecture-based procurement collaborations. Is there any serious industry involvement in this?

I wonder if OWL and friends will eventually obfuscate the real goal; wouldn't a decent media type be a better way to approach this?

Please use 'Hypermedia Constraint'!!

The hit count of this should go to zero! This should skyrocket instead.

Formal Hypermedia Contexts

Formally, a hypermedia context can be understood as a conjunction of predicates that evaluates to true or false for a given URI. The predicates can be expressed as one of the following:

• unary link, e.g service(x)
• binary relationship, e.g. edit-media(u1,x)
• pattern match on representation of x, e.g. r-match( xpathExpr , x)
• pattern match on occurrance of x in representation, e.g match( xpathExpr , x)

Logical sentences build from such predicates can be evaluated against the current state of the client to either test if a given URI matches the denoted context or to find all URIs known to the client that match the predicate conjunction.

The next step will be to express the operations that are currently stated in the form of prose in hypermedia specifications in terms of the above predicate conjunctions and HTTP operations. For example could AtomPub's 'create entry' operation be expressed as the combination of the predicate conjunction match( '/service/workspace/collection@href',x) and POST. Where the match predicate evaluates to true for any URI x that is found as the value of the XPath expression.

This needs more precision of course, but it's a start.

RFC 5545

iCalendar is now an IETF Proposed Standards Protocol (superseedes RFC 2445). Does anyone know whether this is just fixing things or whether there are additional semantics that increase the expressiveness?