Applying the Web to Enterprise IT

Generic Media Types Potentially Leak The Model

2010-11-19T17:12:00.001+01:00

Jean-Jacques Dubray just triggered a thought with a comment he made on Twitter regarding the rough ideas to adopt MVC 'ideas' in JAX-RS 2.0. He was worried that an MVC based approach to creating representations exposes the model to the service consumer.

Interestingly, this is only possible if the service uses generic media types. Specific media types sufficiently limit the kind of information that can be expressed in the representations and prevent the service specific model to be visible to the outside.

Specific media types enforce the resource boundary and prevent the service's domain model from leaking.

This is probably a much better argument against application/xml or application/json than the usual reference to the violation of message self descriptiveness.

REST and the Enterprise

2010-11-15T21:20:00.000+01:00

2010-11-02T01:27:00.000+01:00

"An architectural style called REST (Representational State Transfer) advocates that web applications should use HTTP as it was originally envisioned."

Agency Boundary

2010-10-23T17:00:00.000+02:00

A central aspect of the problem space that REST is designed for is that the communicating components are typically not owned by a common authority and that therefore change must be enabled to happen without coordinating the change between the authorities. For example, for Amazon it is simply impossible to ask any of its customers whether such and such a change 'would be ok'.

In my recent presentation on RESTful HTTP I used the term 'administrative boundary'. Today, I came across the much better term Agency Boundary coined by Rohit Khare and Richard N. Taylor in their ICSE'04 paper Extending the REpresentational State Transfer (REST) Architectural Style for Decentralized Systems:

An agency boundary denotes the set of components operating on behalf of a common (human) authority, with the power to establish agreement within that set and the right to disagree beyond it it.

Interesting Changes Must Surface

2010-10-13T01:55:00.000+02:00

The human brain is truly strange. It happens surprisingly often that you are (well, at least I am) so tied-up in a false assumption that the ability to understand something new is seriously limited.

Then, when the clouding assumption disappears you have these special, enlightening aha-moments.

In this particular case I have been thinking a lot about change impact lately. The false assumption I made was that change would ideally be kept hidden and not surface towards the consumers of some provided capability.

Given that assumption, it seemed strange to maintain a certain RPC criticism I am trying to rationalize: if the goal is to hide change, what is the problem with RPC in the first place?

Then it dawned some minutes ago: Change to a component is essentially driven by requirements to do new stuff and in order to do new things with a component, the changes must surface. Changes that remain hidden are not worth talking about - these are called implementation details. Details that per definition do not matter at the capability level.

Now I can joyfully continue working on my RPC criticism: since changes must surface to be interesting we can compare how they can surface in the various architectural styles.

REST is the only style that requires changes to surface in a way that is guaranteed not to break clients, because REST is rather specific about what can change in which ways. Except for add-on changes (that are not a problem for existing clients) all surfacing changes to a REST server component are bound by the possibilities to vary representations (you may change as long as you do stick to rules of the media type used)

In RPC-based architectures, change is, you guessed it :-), not constrained at all and might surface in a million ways. Welcome to change impact analysis hell...

Artifact Ownership or When Not to Use REST

2010-10-12T00:33:00.000+02:00

Here is an indication of situations when applying REST is not appropriate:
When all artifacts that are affected by a possible change are owned by the same project (e.g. stored in the same source code control repository) then REST is not a suitable style.

An example of this is an application that contains a database component for its private use. Usually the database schema is stored in the repository together with the source code. If you face a need to change one, it is easy to change the other (from a developer coordination point of view).

The general take away from this is that artifact ownership can be used as an indicator for how appropriate a candidate style is.

(Consider how much the Unix command line benefits from the uniform API pipe and filter style: Completely decentralized developers can contribute components (grep, awk, sed, less, sort,..) without even engaging into agreeing on how the components talk to each other). The artifacts that make up the unix tool box are maintained by many different parties all over the world.

Generic vs. Specific Media Types and Evolution

2010-10-12T00:32:00.000+02:00

In addition to the loss of self-descriptiveness there is another problem with generic media types that is not so obvious.

Let's think about the problem from the point of view of someone that needs to conduct a change impact analysis. Suppose there is some REST Web service component C, for example an incident management system, that needs to be changed due to some new business requirement. Also suppose the change applies to the resource implementation that allows for retrieval of incident representations. Specifically, the entity needs to be augmented and changed a little.


GET /incidents/554

200 Ok
Content-Type: application/xml

With the generic media type approach above change impact analysis is quite challenging because it is difficult to figure out, which assumptions the consumers actually make about the entity. Thorough change impact analysis would require the change manager to contact the client owners and investigate their assumptions. Based on that information it is then possible to make changes without breaking existing clients.


GET /incidents/554

200 Ok
Content-Type: application/incident

In contrast, when using a specific media type, there is no need for change impact analysis because the media type itself acts as the contract within which the entity can be changed.

In other words, REST's constraint of specifically typed entities provides for a perimeter within which the server can be flexible without a need to consider the effect the changes have on clients.

Viewing media types as a perimeter for server side change is an important consideration when designing media types. How big you make the perimeter directly determines the bounds within which the server can later on evolve and, of course, how much burden you put on the client for dealing with entities of that media type.

Need For User Involvement vs. General Applicability of Media Types

2010-10-09T03:56:00.000+02:00

The greater the variety of use cases is that a media type can be used to realize the greater is the need for user participation in the implementing application. More steady states are necessary because the user agent cannot differentiate the hypermedia semantics.

Media types with more specific hypermedia semantics support more automatic requests on behalf of the user but are less applicable to general use cases.

In this context it is an interesting observation that HTTP Link headers reduce the need for user involvement while not decreasing the general applicability of the entity.

Should we maybe have a TEST method?

2010-10-05T14:10:00.000+02:00

Just asked myself the question how a service could be put in the position to redirect a test client to a test-version of the URI space it provides?

What about this:


TEST /services/online-shop HTTP/1.1
User-Agent: Simple Tester/1.26

HTTP/1.1 307 Temporary Redirect
Location: /services/test-online-shop 
Content-Type: text/plain

Please go to /services/test-online-shop to exercise tests on this service.

Any service that does not understand what it means to provide a testable version of its capabilities, would just respond with a 501 Not Implemented.

NetBeans 6.9.1, GlassFish 3.0.1 and Jersey 1.5

2010-09-29T23:10:00.000+02:00

Warning:

The procedure described below apparently does not work. I have tried the same thing today (after erasing the .netbeans* directories in my home directory) and there is something missing I think in the 1.4 Jersey library to get this working.

Recently, I have picked up the work on the Jersey View Client module again. Especially I am interested in combining the JAX-RS server side with Jersey's client side framework.

Given that I am really positively impressed by Java EE 6 (thanks to Adam Bien and a bunch of his great presentations) and its integration into NetBeans that natural couple seemed like the way to go.

Turns out it was a lot harder than I thought!

The problem with using NetBeans/Java EE 6 for Jersey View Client development is that there is a really tight (and useful) integration between NetBeans, JAX-RS and GlassFish. Given that Jersey 1.5 is ahead of JAX-RS some work has to be done to get rid of the integrated stuff and add the latest, greatest Jersey builds.

Here is the approach I found to be the least painful:

(This has been tried with NetBeans 6.9.1, GlassFish 3.0.1 and Jersey 1.5)

Install Netbeans 6.9.1

Download Java Version of Netbeans 6.9.1 from netbeans.org and install. You skip this of course, if you have NetBeans already. I just want to describe from a defined starting point and actually re-installed mine to walk through the whole thing.

Remove GlassFish JAX-RS libraries

Remove some libraries distributed with Glassfish as described in 12.2 of Jersey User Guide. The description is not related to NetBeans but the following steps assume you are doing all this inside the GlassFish directory part of your NetBeans installation. For example, /Applications/NetBeans/glassfish-3.0.1/ on my MacBook Pro machine.

Download Jersey 1.5 GlashFish-ready Package

Download Jjersey-gfv3-core-1.5, unzip and copy JARs into the /glassfish/modules/ directory inside the NetBeans-GlassFish installation.

Download Jersey 1.5 View Client Package

Download Jersey View Client 1.5 and install in that same /glassfish/modules directory.

Create a New NetBeans Library for the Jersey JARs

From the Tools menu choose Library and click on New Library to create a new library. Name it restlib_jersey15 (or whatever you like).
Add the following JARs to the classpath by selecting them from the NetBeans-GlassFish modules directory where you have previously copied them to:

jackson-jaxrs.jar

jersey-core.jar

jersey-client.jar

(I am only working on the list now, there are definitely some JARs missing)

I tried to keep it to a minimum to only put those JARs in there that you will need for building. The ones necessary at runtime are in GlassFish already anyway. For that reason, you might need to add to restlib_jersey15 if you have compile time dependencies not being resolved.

Create Example Application

Create new Java Web - Web Application project (with dependency-injection-enabled-box checked if you need it).

In that project, create a new REST resource class: write a class with a method and annotate that with @GET.

For the compiler to be able to resolve the use of @GET you need to add the library you created above to the project libraries folder (right-click on Libraries folder and then choose Add Library.

During the next build, NetBeans should perform its built-in REST integration magic, asking you about the application's resource path configuration. Use the default action and click OK.

The build process triggered by this will pull in some other REST library (I have not been able to figure out where to disable that). Simply remove those and they usually do not get added again.

Now you are all set to start playing with Jersey Client Views.

Notes

There is probably more that can be done. Especially there seems to be a way to overwrite the REST-related project properties to directly use the library you created instead of the JAX-RS 1.1. one. However, after one day of trying, I gave up. It seems to be hard wired at some point inside NetBeans. Any help on this would be greatly appreciated.

If you find I made any mistakes or if it works differently for your setup, please let me know.

2010-09-25T18:28:00.000+02:00

Download all of jersey at https://jersey.dev.java.net/svn/jersey/
import into netbeans
open projects
core
server
client
experimental/view-client

clean and build explicitly

Spotted Alternates Header in the Wild

2010-09-01T02:33:00.000+02:00

Just spotted an Alternates header in the wild:


$ curl http://www.w3.org -HAccept:text/foo -I

HTTP/1.1 406 Not Acceptable
Date: Tue, 31 Aug 2010 22:26:53 GMT
Server: Apache/2
Alternates: {"Home.html" 1 {type text/html} {charset utf-8} {length 28437}}, {"Home.xhtml" 0.99 {type application/xhtml+xml} {charset utf-8} {length 28437}}
Vary: negotiate,accept
TCN: list
Connection: close
Content-Type: text/html; charset=iso-8859-1

GET /stock-quote/foo vs. getStockQuote("foo")

2010-08-25T01:46:00.000+02:00

Consider the good old stock quote example from REST vs. RPC discussions. In both variants two kinds of coupling exist.

On the one hand there is the intentional coupling that causes the client to make the call to that particular remote thing and not just any arbitrary one. The intentional coupling is a human choice, manifested in configuration or code.

On the other hand there is technical coupling because the client software needs to know (aka be coupled to) the provided interface. Otherwise the communication would no be able to happen.

Given those two kinds or layers of coupling it makes no sense whatsoever to repeat the specifics already present in the intentional coupling at the technical level by giving that specific remote thing a specific interface.

Once the intentional choice has been made to talk to that remote thing the interface specifics can be factored away.

Why design, implement, test, maintain, document and explain more stuff if you can do the same things with less?

PUT and Content-Location

2010-08-23T02:10:00.000+02:00

I noticed some days ago that I totally missed the use of the Content-Location header in responses to PUT requests. Used in this scenario, the client does not need an additional GET to obtain a representation of the resource after the PUT.

Caches can also cache the response of the PUT in this case.


PUT /foo
Content-Type: application/order


  ...


200 Ok
Content-Location: /foo
Content-Type: application/order
ETag: "1"


  accepted
  ...

The Content-Location header is telling the client: "This is the entity and entity headers you would receive for a GET to /foo".

How much machine do you need?

2010-08-16T16:37:00.000+02:00

Thoughts on Media Types

2010-08-16T11:42:00.000+02:00

Amazingly Close

2010-05-21T00:59:00.000+02:00

While looking up some stuff in the book "Integration Patterns" I spotted a tiny sentence that is amazingly close to REST. In chapter one on page 3 the authors (among them Gregor Hohpe, BTW) emphasize that from the point of view of integration the whole must be considered instead of solely designing for each integration between two applications in turn. The sentence is

However, if you approach this same problem from an integration architecture perspective, the ideal application is a thin layer of presentation that consumes shared functionality or data at the enterprise level.

I find this amazingly close to the notion of a user agent component consuming representations and capabilities provided by resources.

Serendipitous Reuse vs. Reusable Services

2010-05-12T10:51:00.000+02:00

The notion of reusable service builds on the anticipation of its uses

Reusable components notion focusses on using the same component for the same purpose often. There is no focus on use in unanticipated contexts. seren. reuse oth does that.

Serendipitous reuse on the other hand focusses on building components (that expose functionality) and using those compoents in unanticipated ways.

IRC Conversation on User, User Agent, Media Types, Application etc.

2010-05-06T11:13:00.000+02:00

Currently I am trying to figure out the significance of the various aspects of a RESTful architecture with regard to modeling. Such aspects as user agent, media type, steady state, user, application, and application state. Tried to build-up an explanatory train of thought on that basis in an exchange with Philipp Meier on #rest IRC yesterday. It is the typical, hard to read, IRC conversation but you might find it useful nevertheless.

Jersey Client Side: The ViewResource Class

2010-04-30T13:35:00.000+02:00

The last posting in this introductory series provided a high level overview of the Jersey client side framework I am working on together with Paul. In this one, we look at some of the details.

At the heart of the framework lies the concept of ViewResources. Conceptually, ViewResources are an extension of the existing Jersey client side WebResource class. A ViewResource is a client side 'handle' for an HTTP resource through which HTTP requests to the resource can be performed. While WebResource 'invocations' result in ClientResponses or object representations of the response entities 'invocations' of ViewResources produce view objects that encapsulate assumptions (including anticipated next transitions).

The choice of view class for a given request completely expresses the assumptions made about the resource and the response. The view class in turn encapsulates these assumptions in a single artifact.

Regarding the examples that follow it is important to keep in mind that the view objects are not to be seen as local proxies for remote 'objects' but that the view objects hold application state and provide access to the data contained and a means to activate the controls that are (maybe) embedded in it.

In a subsequent installment of this introductory series we will see how even the entities of error responses constitute application state and can be used to construct corresponding views (possibly with contained application data and/or embedded controls for outgoing state transitions).

GET requests


URI poUri = URI.create("http://foo.org/purchaseOrders/42");
ViewResource vr = client.viewResource(poUri);
PurchaseOrderView pov = vr.get(PurchaseOrderView.class);

For GET requests there is a shorthand removing the need for an intermediate ViewResource object:


URI poUri = URI.create("http://foo.org/purchaseOrders/42");
PurchaseOrderView pov = client.view(poUri,PurchaseOrderView.class);

PUT requests


URI poUri = URI.create("http://foo.org/purchaseOrders/42");
ViewResource vr = client.viewResource(poUri);
PurchaseOrderView pov = vr.put(PurchaseOrderView.class, "some purchase order data");

For the use with PUT requests the build method of the PurchaseOrderView class could be implemented to work with both 200 Ok and 204 No Content responses. In the former case the body woud be interpreted as a representation of the order and in the latter the build method could perform an additional GET on the request URI to obtain a a representation of the order. We will discuss such support for automated transitions in a dedicated posting.

POST requests


URI colUri = URI.create("http://foo.org/purchaseOrders");
ViewResource vr = client.viewResource(colUri);
CreatedView cv = new CreatedView(PurchaseOrderView.class);
vr.post(cv, "some purchase order data");
PurchaseOrderView newPoView = cv.getCreated();

This example makes use of the 'helper' view class CreatedView<T> provided with the framework. This view class encapsulates the HTTP mechanics related to 201 Created responses and makes the created resource retrievable as a view through the getCreated() method.

Note how this example instantiates the view class outside the framework and supplies the instance directly to the post() invocation.

DELETE requests


URI poUri = URI.create("http://foo.org/purchaseOrders/42");
ViewResource vr = client.viewResource(poUri);
VoidView vv = vr.delete(VoidView.class);

This example uses the framework supplied view class VoidView to handle the (possible empty) response to the DELETE request. Because the VoidView really does not express any kind of expectation its use outside the context of pure examples is rather useless.

Expectations pertaining to the context of a request should be encapsulated inside a view class and not be propagated into the code that actually uses the view. Deal with HTTP mechanics (such as checking response status) inside view classes.

A RESTful Client Side Framework for Jersey

2010-04-29T16:24:00.000+02:00

For the past couple of weeks Paul Sandoz and I have been working on a client side framework for Jersey that encourages RESTful design on the client side. After a trial period with frequent code changes, we feel that the code base is now stable enough to go public.

The framework introduces an annotation based approach that enables developers to turn arbitrary classes of the client side code into HTTP response handlers. The framework refers to instances of such classes as 'views' (mostly for the lack of a better term).

Views serve three purposes:

1. Build-up state corresponding to the handled response

View classes provide specifically annotated methods that will be invoked by the framework core to handle the response. These 'build' methods are intended to build-up the state of the view object from the HTTP response.

2. Present or make accessible the data of the current application state
Usually, users (human or machine) intend to use the information contained in HTTP response bodies. View objects make this information accessible either through GUI elements for human users or through access methods for machine users.

3. Use the hypermedia controls embedded in the response body to turn user (human or machine) actions into HTTP requests.

Both human and machine users interact with RESTful applications through user agent components by activating the controls embedded in the representations of the current application state. View objects provide a means for the user to activate the controls and will then create the corresponding HTTP requests. The means for control activation can be GUI elements in the case of human users or regular methods in the case of machine users.

Enough theory - let's look at some code.

Suppose we want to develop a view class that is is suitable for Atom Publishing Protocol collections. The following examples step by step show the different aspects of such a view class and how it could be used.

First, we need a build method for handling HTTP success responses to GET requests to collection resources.


public class CollectionView {

	private Feed feed;

	@GET
	@Status(200)
	@Consumes("application/atom+xml")
	public void build(Feed feed) {
		this.feed = feed;
	}
}

The annotations on the build-method indicate to the framework core that the method applies to responses to GET requests that have a status of 200 Ok and a response body of the media type application/atom+xml. As requested by the method parameter list the framework core will inject the response body as a Feed object to enable access to the body inside the build method.

The code below shows how the CollectionView class might be used. The new view method on the Jersey Client class performs an HTTP GET on the provided URI, instantiates a new object of the provided class and then dispatches to a suitable build method of the created instances (based on the annotations).


public void main(String args[]) {
	URI uri = URI.create("http://example.org/blogs/jim/entries");
	Client client = new Client();
	
	CollectionView cv = client.view(uri, CollectionView.class);
}

Next we need to add to our CollectionView class methods that provide access to information contained in the feed. To keep things simple, we'll just provide a getter for the title of the feed.


public class CollectionView {

	private Feed feed;

	@GET
	@Status(200)
	@Consumes("application/atom+xml")
	public void build(Feed feed) {
		this.feed = feed;
	}

	public String getFeedTitle() {
		return this.feed.getTitle();
	}
}

Example for using the getFeedTitle() method:


public void main(String args[]) {
	URI uri = URI.create("http://example.org/blogs/jim/entries");
	Client client = new Client();
	
	CollectionView cv = client.view(uri, CollectionView.class);
	System.out.println("Feed title is " + cv.getFeedTitle());
}

The third purpose of view classes mentioned above is to provide users (human or machine) with a means to activate the controls embedded in the response entities. RFC 5005 defines (among others) the 'next' link relation that can be used to tell a consumer of the feed document where the next page of the feed representation is located. We will use this link relation to demonstrate how a view class can expose a hypermedia control to the user.

One possibility to make 'next' links accessible to the user is to provide some getNextPageUri() method on the view class that can be called by the user to retrieve the URI of the next page. However, we can also provide a design that matches better the usual intent of the user of a feed: to iterate over the contained entries.

The code below shows an iterateEntries() method that processes all entries in the feed document of the view and follows any 'next' link to subsequent feed pages. The EntryHandler interface acts as a callback class.


public interface EntryHandler {

	public boolean handleEntry(Entry e);

}


public class CollectionView {

	private Client client;
	private Feed feed;

	@GET
	@Status(200)
	@Consumes("application/atom+xml")
	public void build(Feed feed, @Context Client client) {
		this.client = client;
		this.feed = feed;
	}

	public String getFeedTitle() {
		return this.feed.getTitle();
	}

	public void iterateEntries(EntryHandler entryHandler) {

		List entries = this.feed.getEntries();
		for (Entry entry : entries) {
			boolean proceed = entryHandler.handleEntry(entry);
			if (!proceed) {
				return;
			}
		}

		Link link = this.feed.getLink("next");
		if (link == null) {
			return;
		}
		
		URI nextUri = link.getHref().toURI();
		CollectionView v = this.client.view(nextUri, CollectionView.class);

		v.iterateEntries(entryHandler);
	}

}

Note that we have added an annotated parameter to the build method to inject the client:


public void build(Feed feed, @Context Client client) {

This is necessary because the iterateEntries() method constructs a view for the next page if a 'next'-link is found.

The CollectionView class can now be used to iterate over all entries in a feed as follows:


public void main(String args[]) {
	URI uri = URI.create("http://example.org/blogs/jim/entries");
	Client client = new Client();
	
	CollectionView cv = client.view(uri, CollectionView.class);

	 cv.iterateEntries(new EntryHandler() {

		private int count = 0;

		public boolean handleEntry(Entry entry) {
			this.count++;
			System.out.println("Entry #" + this.count + ": " + entry.getTitle());

			// limit the number of processed entries to 200
			return (this.count <= 200);
		}
	});

}

(You can find a similar example in the Jersey sources at experimental/view-client/jersey-view-client-samples/atompub-simple-view-client-sample)

The CollectionView example class demonstrates the three purposes of a view:

Handle HTTP responses

Present or make accessible application state

Provide a means for the user to activate hypermedia controls

Interactions with resources never occur without assumptions about the resources (for example that it is a Web page, an image or an Atom Publishing Protocol collection). View classes are manifestations of such assumptions and the choice of a specific view class for a given interaction expresses which assumptions are being made.

From a code structuring and maintenance perspective view classes nicely bundle up all the elements that pertain to such assumptions and act as a single point of documentation.

There are a number of issues we have not covered in this first blog. Among them are

Support for several response body media types

Error handling

Constructing requests other than GET

Performing a sequence of automated requests

Using 'existing' client side classes as view classes

We will talk about those and other features in follow up postings.

Instructions for Obtaining the Code

Unfortunately there are currently issues obtaining the latest SNAPSHOT artifacts from the java.net maven repository. Until this is resolved it is necessary to obtain and build Jersey yourself. The new client side framework is located in experimental/view-client.

Use SE 6 with maven version 2.2.0 or 2.2.1.

Join the Jersey project as an Observer at https://jersey.dev.java.net.

Check out the trunk source:


svn checkout \
https://jersey.dev.java.net/svn/jersey/trunk/jersey \
jersey --username <username>

Clean and install:

mvn clean install

If you want to do this a bit quicker you can skip the tests

mvn -Dmaven.test.skip=true clean install

You may need to set the following maven options (which can be set using the MAVEN_OPTS environment variable):

-Xmx1048m -XX:PermSize=64M -XX:MaxPermSize=128M

Discussion

Please send comments to the jersey users list.

Steady-State

2010-04-04T20:29:00.000+02:00

I have been meaning to write up for some time something about the notion of steady state. Now it ended up in a rest-duscuss posting.

Some references that I found insightful:

Steady-State in Roy's dissertation

Posting dealing with application entry points

Web applications can have many entry points

Steady-states are to be understood in isolation

Steady-State, bookmarks, cool URIs(last two paragraphs)

Slightly related from this blog: In Fear of Sub Requests.

Why 'Action Resources' are a REST Anti Pattern

2010-03-31T14:25:00.000+02:00

REST's uniform interface constraint requires the operations that can be invoked on a resource to be generic, meaning that the applicability of an operation must not depend on the actual nature of the target resource. The uniform interface does not prohibit additional methods to be defined but requires any extension method to be generic. PATCH or MONITOR (slide 16) for example are valid extensions, while ORDER or PAY are not.

Our OO-biased brains are trained to think in terms of classes and associated operations (Cart.order()) and it apparently takes a considerable amount of time for our brains to re-wire and think in terms of transferring representations to modify resource state.

As a result, people are tempted to come up with ways to map non-uniform operations onto HTTP's uniform interface. One offspring of such endeavor is the REST anti pattern of Action Resources.

I have not tracked it back to its origin, but the general form goes something like this:

Given an operation foo(), define a link semantic 'foo' that enables the server to tell the client what the URI is of 'the foo-action resource of some other resource R. Knowing the foo-action resource R_foo, the client would then be able to invoke the foo() operation on R by means of an empty POST to R_foo:

Find foo-action resource of R:


HEAD /items/56

200 Ok
Link: ;rel=foo

Invoke foo() on R:


POST /items/56/foo
Content-Length: 0

204 No Content

So, why am I calling it an anti pattern?

Action resources are an anti pattern because the approach violates REST's self descriptive messages constraint. How so? Because the meaning of the POST request depends on resource state at the time the client learned that /items/56/foo is the foo-action resource of /items/56. There is nothing in the request that allows the server to understand the actual intention of the client at the time the server handles the POST request.

Suppose the client issues the above HEAD request at time T₁, the server replies at T₂ and the client receives the response at T₃. By the time T₄ the client sends the POST request to the action resource (and T₅ when the server actually receives it) the server might have changed and is now looking at the POST with the empty body which translates to the client intention of telling the resource /items/56/foo to process this [empty body].

The server does not know that the request semantics depend on the server state at T₂ when the server created the HEAD response and therefore cannot detect any mismatch between client intention and its own interpretation.

DELETE and 201 Created

2010-03-07T15:31:00.000+01:00

Just thought about a useful combination of DELETE and a 201 Created response:


DELETE /service/documents/667

201 Created
Location: /service/archive/documents/667
Content-Type: text/plain

Document /service/documents/667 has been deleted
and archived at /service/archive/documents/667.

The Farmer in the Dell

2010-03-02T15:41:00.000+01:00

A couple of minutes ago, Bob Haugen posted a question on the rest-discuss list that essentially comes down to the question of how the use of REST affects a classic 2PC scenario. Note that the question is not how you do 2PC with REST but what the implications are regarding compensation strategies etc.

The scenarios is this: There are N farmers that each have an amount of X_n pints of berries to sell. The farmers tell M market places about their amount and K buyers order berries from the market places. The 2PC issues come in when orders to different market places overlap and more berries are ordered from a farmer than he has available.

I tink this relates very nicely to my RESTifying Procurement endeavor and I therefore plan to make the scenario the overall target to which my experiment should evolve.

Classifying the CouchDB API

2010-02-27T18:16:00.000+01:00

In the context of my Classification of HTTP-based APIs @hanonymity today asked me, how I would classify the CouchDB API.

Ok, let's see. Going to the HTTP Document API immediately reveals that the API definitely violates the hypermedia constraint (see last paragraph) because there is an API documentation in the first place. The only thing one would expect to see for a RESTful API is a set of media type specifications along the lines "The CouchDB API uses the following media types and link relations....which are specified here...".

Next, let's check if the API can be classified as HTTP-based Type II. The fastest way to verify this is usually to look for the use of only specified media types and it is immediately obvious that the CouchDB uses the generic media type application/json and not a specific one that would make the messages self-descriptive. CouchDB API fails the test for HTTP-based Type II, too.

This leaves us with the question whether the API is HTTP-based Type I or if we have to let go all hope because it must be classified as RPC URI-Tunneling. The thing to look out for is of course the use of action names in URIs. It does not take a lot of browsing through the API documentation to reveal that the CouchDB API designers knew what they were doing. The API very thoroughly leverages HTTP mechanics and we can happily conclude that the API is an HTTP-based Type I API.

Is it a problem that the CouchDB API violates two out of four of REST's interface constraints and is therefore not REST at all? I do not think so, because I would not consider achieving loose coupling between a database (backend) and the component that uses the database to be a very useful goal. At least not at the cost that you have to pay on the client side and also because there is strong coupling around the schema anyway between a database and the code that is using it.

However, I think CouchDB API shows quite nicely how an API can still benefit from the simplicity induced by HTTP-based Type I even if we cannot label the API as REST.

A note on the COPY method: It would be helpful to say in the API documentation that the COPY extension method is actually WebDAV's COPY method. And while we are at it, it makes also sense to note that COPY does not really fit HTTP because COPY is a method that works on two resources (Source and Destination) while HTTP does not support such method semantics. For example, caches would not understand that they need to flush the representations of the (now overwritten) destination resource.

This is not a question of RESTfulness though. It would be entirely possible to design an architecture that adheres to the REST style and provides methods that work on two resources.

The HTTP-API Family

2010-02-25T11:02:00.000+01:00

fixme

.. a classification of web apis to differentiate from rest and to make explicit their properties

and what is saved by using them

Service Types Revisited

2010-02-15T02:07:00.000+01:00

Working on the RESTifying Procurement show case I realized that it looks as if I had to revisit my approach towards service types. I have argued that a service type is constituted by the set of hypermedia semantics it makes use of. This seemed reasonable since a client developer needs to know at least a minimal set of the possible hypermedia semantics to expect from a service in order to write a client for a service of that kind.

Unfortunately this approach has some problems when services of different kinds use the same set of hypermedia semantics because the differentiating aspect is lost. I realized this because in the procurement example I am basically using a single media type but still have a range of services, for example supplier or carrier.

A possible solution to this issue is to have the all-encompassing media type define the service types. Such types are still necessary to enable lookup based on type, for example in order to find the carrier service of some external business partner. Looking for the procurement service doesn't make that much sense.

Markus Karg on the Hypermedia Constraint

2010-02-14T18:53:00.000+01:00

As a reaction to the latest experimental proposal for supporting hypermedia in Jersey, Markus chimed in, saying:

If you want to gain all the benefits of REST, you need to apply all four constraints but not just three of them.

I whole-heartedly agree with that.

However, I disagree with his analysis that only the response body (and not the headers) carries the hypermedia that represents the next application state. From a hypermedia constraint point of view it is irrelevant whether the hypermedia controls (that is links, link templates or forms) are placed inside the body or inside the HTTP headers.
Placing the controls into the HTTP header can, for example, be a viable solution when you want to (or have to) use existing document formats for the body that do not provide the ability to place the controls in the body.

Markus Karg on the Hypermedia Constraint

2010-02-14T18:45:00.000+01:00

As a reaction to the latest experimental proposal for supporting hypermedia in Jersey, Markus chimed, in saying:

If you want to gain all the benefits of REST, you need to apply all four constraints but not just three of them.

I whole-heartedly agree with that. In fact, I think that the application of REST in contexts outside the Web is in a rather vulnerable state because of the increasingly advertised misconceptions about REST.

Applying REST to a problem space leads to significantly higher start-up efforts than solving the same problem for the given moment in time with any RPC technology. The reason to apply REST is to leverage the long term benefits induced by REST's architectural constraints and these benefits can only be leveraged if you apply all four of the constraints and not if you drop the hypermedia constraint. To quote Roy:

Umm, no, it is essential to eliminate the coupling between client and server. If the application doesn't follow the workflow defined by the representations that are received, then the application isn't using the REST style. Not even a little bit. It is using RPC plus streaming, with a rather inefficient syntax, and the client will break each time the server's application evolves because the client must be anticipating the server's state based on its own assumptions. In other words, the two are coupled by their original design.

(From rest-discuss #8380)

Applying the hypermedia constraint has the implication that the client really must prepare for any kind of response by the server instead of relying on some undocumented out-of-band agreement between client and server. Maybe this implication seems so radical that

Three Aspects of Steady-States

2010-02-13T12:48:00.000+01:00

The #rest IRC channel (transcripts here and here) has recently become for me a valuable source for thought stimulation (come and visit). Yesterday we had a discussion regarding steady-states and the following observation has been made:

URIs refer to a certain application state; at least in the sense that one can use a URI to go back to a certain application state or that one can pass a URI to another party to bring this party into that application state.

However, 'that application state' (despite the notion that I can use the URI to get back to it) is not stable over time. The semantics of the mapping are, but the transitions available form that state can change. So, what is the significance of 'that application state'?

I have not figured that out yet, but here is a thought I had this morning as a reaction to the discussion:

A Web application comprises a state machine that can change over time. Each state (aka steady-state) of that state machine has three aspects:

Its semantics (what the state means)

The serialized state of the associated domain concept (e.g. order 10029)

The outgoing transitions to other states

The first of the three is constrained to remain stable over time the other two vary depending on the state of the associated domain concept and the actual state machine the server intends to provide to (that particular) client.

UPDATE: It has come up in a number of places a notion of transient states or ephemeral URIs. As far as I understand the issue circles around the idea of giving distinct URIs to different states of application states. With this approach, an order in some 'review-pending' state would be represented by a different resource (and hence different URI) than the same order in the state 'shipment-initiated'. Please correct me, if I miss the point here.

My response to that can be found in this comment on Ian's blog.

I'd highly apprechiate if someone could shed more light on this issue. I think it is a deep one.

RESTifying Procurement - Service Running

2010-02-12T12:23:00.000+01:00

Though not much yet, I have put the procurement service online. At the moment, the service for the supplier party of the UBL business process is provided by the implementation. It can be accessed at http://labs.nordsc.com:8080/rp/supplier/service.


$ curl http://labs.nordsc.com:8080/rp/supplier/service | tidy -xml -i -q

xmlns:atom="http://www.w3.org/2005/Atom">
  
    This service implements the UBL
    supplier role.
        href="http://labs.nordsc.com:8080/rp/supplier/provider/common-catalogue"
    type="application/procurement+xml;type=catalogue"
    title="The public catalogue">

The service document provides a link to the common catalogue provided by the provider role (here played by the supplier party). I explain the special use I make of AtomPub service documents in this post.

A client that understands the 'common-catalogue' link relation and intends to look at the common catalogue can now do this:


$ curl http://labs.nordsc.com:8080/rp/supplier/provider/common-catalogue | tidy -xml -i -q



xmlns:cbc="urn:oasis:names:specification:ubl:schema:xsd:CommonBasicComponents-2"
xmlns="urn:oasis:names:specification:ubl:schema:xsd:Catalogue-2">
  public-001
  2012-02-02
  
    1
    
      
        16.90
      
    
    
      Valve 1-54
      This valve helps with a lot of
      things
      
                schemeDataURI="http://labs.nordsc.com:8080/rp//provider/items/">
        PARTS-7615221
      
    
  

    2
    
      
        26.90
      
    
    
      Valve 2-54
      This valve helps with a lot of
      things
      
                schemeDataURI="http://labs.nordsc.com:8080/rp//provider/items/">
        PARTS-7625222
      
    
  
  
    3
    
      
        36.90
      
    
    
      Valve 3-54
      This valve helps with a lot of
      things
      
                schemeDataURI="http://labs.nordsc.com:8080/rp//provider/items/">
        PARTS-7635223

According to the to-be-written processing intent of application/procurement+xml a client could now select items from the catalogue and create an order from them. How that order looks and - more interesting - where to send that order will be the subject of one of the next posts.

The implementation will serve the examples as application/xml to browsers thanks to Jersey's conneg support so you won't get an annoying download. There is also CSS styling of the XML so you can actually see something.

RESTifying Procurement - The Media Type

2010-02-12T02:44:00.000+01:00

In the examples throughout this series I am using (among others) the hypothetical media type application/procurement+xml. In this post I will start fleshing out its specification.

A central aspect of the RESTifying Procurement project is the use of the UBL document family. UBL actually encourages adopters to customize the documents to suit the needs of the interacting parties which usually means specializing upon the very flexible UBL documents types. There are also guidelines for evolving the document types to forward incompatible new types but I'll rather stick with standard UBL.

UBL has not been build with hypermedia in mind and does not provide explicit linking facilities. In order to address that limitation I'll define a special processing intent for <cbc:ID> elements, make use of HTTP Link headers and, where appropriate, use hypertext enabled media types such as application/atom+xml.

The hypothetical media type application/procurement+xml encompasses specialized versions of several UBL document types and associates with them a processing model as far as that is necessary to enable the associated business processes to be carried out by software components communicating via HTTP.

The current list of document types (which I will update when new types are added) is shown below. For every document type a value for a type parameter for the media type is provided to enable content negotiation based on the individual document types.

UBL Document	type parameter value	RelaxNG Schema	Example
Catalogue	catalogue	Catalogue.rnc	(xml)
Order	order	Order.rnc	(xml)

Additional Processing Rules for the <cbc:ID> Element
The <cbc:ID> element is used throughout the UBL for identification. Various attributes can be used to identify the schema to which the identifier itself belongs. A common use of the element would be




  PARTS-7635223

Which means that the thing being identified has the identifier PARTS-7635223 and that this identifier is taken from the scheme (namespace) identified by http://labs.nordsc.com/schemes/products. This is similar to Atom's <category> element that combines a scheme and a term attribute for the same purpose.

Unfortunately, the element does not directly provide for the use of URIs for identification, meaning that one cannot do something like this:

That is a strange situation (at least in Web context): UBL provides an element for identification but does not support the use of URIs as identifiers. To solve this, I decided to cheat a little and turn the <cbc:ID> element into a hyperlink.

The <cbc:ID> element provides an attribute named schemeDataURI that enables authors to reference a resource where the scheme data is available (the list of all identifiers and their description).

What if I defined the processing intention of application/procurement+xml to understand the concatenation of the schemeDataURI and the actual identifier to be a URI identifier for the thing identified by the given <cbc:ID> element? Yes, that would enable <cbc:ID> to turn into a hyperlink.

An element such as



  schemeDataURI="http://labs.nordsc.com/service/items/">
  PARTS-7635223

can then be interpreted as a link referencing http://labs.nordsc.com/service/items/PARTS-7635223.

For now the specification of application/procurement+xml consists of the specialized Catalogue document type and the special processing rule for <cbc:ID> element.

I will update this posting as the media type grows.

More Comments on Jersey

2010-02-10T13:22:00.000+01:00

This posting to the Jersey user's list provides more detailed comments, explaining my concerns.

UPDATE: The thread started by the reference above turned out to be rather interesting. It's worth a look.

Jersey Going Astray?

2010-02-09T22:49:00.000+01:00

Santiago Pericas-Geertsen today wrote about the approach currently taken by Jersey towards providing active server-side support for the hypermedia constraint.

Given the exposure of Jersey I am worried to say the least.

Santiago writes:

It has been identified by other authors that there are actions that cannot be easily mapped to read or write operations on resources.

This quote creates the impression as if there was a problem that needed to be solved and that Jersey will help developers to solve it. This scares me!

It scares me because it looks as if one of the most exposed RESTful HTTP frameworks is about to evolve not on the basis of how to enforce RESTful systems but based on a likely misconception of REST. What we need is to help architects and developers to design RESTful REST systems and not support REST misconceptions by hard coding them into a framework. Apologies if this sounds insulting, I don't mean to be. I am just worried.

Here is the misconception: with REST, we do not map actions to read or write operations on resources but we achieve coordination between processes by transferring representations. Representations sent from server to client advance the client's application state and representations that are sent from client to server change resource state. The notions of 'action' and 'read or write' resources simply do not exist in REST and just because some authors have identified a problem related to these notions should not drive the evolution of JSR311.

In REST a coordination goal is achieved by changing the state of a resource that has the proper semantics. All that is needed is the specification of media types that provide the necessary hypermedia semantics for the client to find such proper resources. RESTful frameworks should encourage the design of proper media types and ideally enforce correct client side behavior instead of introducing a concept of 'action resources' that just does not exist in REST.

The primary design artifact for RESTful services is media types, link relations, etc. (there just is nothing else to design) and behind those is very serious and time consuming design work. Pretending that a framework (let alone 'action resources') could make this any simpler is misleading.

I guess I have now put myself in a position of having to provide more constructive feedback :-) I'll give that a try tomorrow.

And apologies again to those people whose work I am criticizing.

UPDATE: Markus Karg chimes in.

RESTifying Procurement – Sourcing Part 2

2010-02-09T12:29:00.000+01:00

In this part of the procurement series I describe the minimal UBL Catalogue document subset used for RESTful sourcing. In the previous part I have already presented the minimal HTTP interaction to send a public catalogue to a client and in this posting I'll show what information the catalogue needs to contain to enable the client to perform the next procurement step: ordering.

The intention of a catalogue is to inform the receiver what items are available and to what prices and conditions. This information is then used by the receiver when constructing an order. One way to implement this pattern would be to make the catalogue a form, let the receiver select items within the form and finally let the receiver place an order by submitting the form. However, I'll have to take a different approach because on the one hand, there might be multiple catalogues involved in the 'shopping process' before the ordering and on the other hand, a forms based approach is not possible when using UBL documents because they do not provide forms.

The UBL based solution is to list available items in the catalogue document and to provide SellersItemIdentification information for each item that the receiver can use to construct the order. This is essentially a client-side shopping cart pattern in which the client collects item identifiers and then submits the collected set as an order.

Here is the UBL catalogue (I did not define a schema, the intention is that the example document shows the possible elements):


GET /provider/common-catalogue
Accept: application/procurement+xml

200 Ok
Content-Type: application/procurement+xml
ETag: public-001
Cache-Control: public,must-revalidate



xmlns:cbc="urn:oasis:names:specification:ubl:schema:xsd:CommonBasicComponents-2"
xmlns="urn:oasis:names:specification:ubl:schema:xsd:Catalogue-2">
  public-001
  2012-02-02
  
    1
    
      
        16.90
      
    
    
      Valve 1-54
      This valve helps with a lot of
      things
      
        PARTS-7615221
      
    
  
  
    2
    
      
        26.90
      
    
    
      Valve 2-54
      This valve helps with a lot of
      things
      
        PARTS-7625222
      
    
  
  
    3
    
      
        36.90
      
    
    
      Valve 3-54
      This valve helps with a lot of
      things
      
        PARTS-7635223

Let's analyze that in chunks. Here is the top:




xmlns:cbc="urn:oasis:names:specification:ubl:schema:xsd:CommonBasicComponents-2"
xmlns="urn:oasis:names:specification:ubl:schema:xsd:Catalogue-2">
  public-001
  2012-02-02

The top of the document contains UBL-specific namespace declarations and a catalogue ID and issue date. For debugging purposes, it is useful to provide a style sheet for rendering the catalogue in a browser, so an xml-stylesheet PI is included also.

The catalogue consists of catalog lines that each contain information of one offered item:



    3
    
      
        36.90
      
    
    
      Valve 3-54
      This valve helps with a lot of
      things
      
        PARTS-7635223

This example catalogue line describes the item Valve 3-54 is available at a price of EUR 36.90 and that the item is identified in the seller's system by the ID PARTS-7635223. This seller's item identification is what the buyer later on must use for constructing its order.

How does the client know that? This is information that needs to be covered by the media type specification. Therefore, the specification of the hypothetical media type application/procurement+xml would need to specify that catalogue documents' catalogue lines contain seller system IDs and that these must be used in the corresponding fields of order documents.

This highlights the distinction between document schema and intended processing and why it is not enough to send the catalogue as application/xml, for example. It is exactly the hypermedia model (e.g. "take the seller's item IDs and put them in your order later on") that is defined by the media type and cannot be defined by a schema.

Why do I use a common media type for all UBL documents (application/procurement+xml) as opposed to one media type for each UBL document (e.g. application/ubl-catalogue+xml)?

A single generic media type prevents the creation of possibly many specific types; this in my experience simplifies development

Media types commonly identify the application that should be used to process them, so it seems consistent to think of an application that handles procurement related documents and figures out internally which specific document it has been given.

A type parameter can be used for content negotiation and related aspects such as declaring what format to send to certain resources. For example, in order to tell a client that an AtomPub collection accepts orders <accept>application/procurement+xml;type=Order</accept> does the job.

There is another aspect of the media type design which I will cover in one of the next parts: The primary purpose of an HTTP response is not to transfer a serialized domain object (e.g. a catalogue) but to put the client in a certain application state. Therefore, media type design must be primarily concerned with the question of what the intended application state is. The decision to make use of UBL for the procurement example series limits the design space because I cannot simply add necessary control data and/or links to UBL.

The way to solve this problem is to make use of HTTP Link headers to provide clients with information such as where to send an order once it is constructed.

REST Web services with Yaws Yapps

2010-02-08T11:58:00.000+01:00

INTRO

First, you need to build and install yapps because it is not installed during the yaws standard build and install.


$ cd yaws-1.87/applications/yapp
$ make
$ make install

Then Yaws needs to be made aware of the yapp module. I followed the instructions in the yapps intro. This worked pretty much, but I had to look up the yapp's ebin directory from the yapp Makefile beause it was not below Yaws' root directory as the intro suggests.

yapps introduction

Setting up Erlang, Yaws and CouchDB

2010-02-07T22:00:00.000+01:00

First I thought I'd develop the practical examples for my RESTifying Procurement experiment on the basis of Jersey but fortunately before I started I realized the obvious opportunity to dive into Erlang and CouchDB. Hence here we go.

This is a very brief list of more or less painful steps I had to go through to set up a basic environment on the server hosted by my ISP:

Download latest Erlang sources from erlang.org

Run configure script...to find out that my ISP provides only a considered-broken gcc with its SuSE distro and updates

Download newer, fixed gcc sources and compile fixed gcc with broken gcc

Compile Erlang with fixed gcc (smooth compile and install now)

Download Yaws

Compile, install, configure and start Yaws (super easy)

Download CouchDB

Compile, install CouchDB

In between I had to add PackMan's SuSE repository to yast to get hold of some packages that are apparently unknown to my ISP's SuSE repository and figuring this out was rather time consuming because I usually do not spend much time with SuSE administration issues these days.

I had two Erlang books (Erlang Programming and Programming Erlang) waiting on my shelf for me and spent my (little) free time over the weekend to get my feet wet. So far, I am mostly fascinated because I have never done functional programming before.

What I found kind of difficult was to find documentation and code snippets and decide about their quality and whether they reflect current practice. Here is the list of links I have dug up so far and consider prime content:

The Erlang Web site

The Yaws Web site

RESTful Web services with Erlang and Yaws (Steve Vinoski)

Build you next Web application with Erlang (Dave Bryson, Steve Vinoski)

A RESTful we service demo in yaws (Nick Gerakines)

Steve Vinsoki's blog

Nick Gerakines' blog

Erlang - software for a concurrent world (Presentation by Joe Armstrong)

Systems that never stop (and Erlang) (Presentation by Joe Armstrong)

(UPDATE) OTP Design Principles User's Guide

That's it for now. When I dig up more articles I'll append them here.

RESTifying Procurement - Sourcing Part 1

2010-02-04T16:07:00.000+01:00

Sourcing refers to the procurement process where one party role (the provider) provides another party role (the receiver) with a catalogue. A catalogue is a document that describes items and prices.

The first thing to do is to differentiate server side role and client side role to achieve clean separation of concerns. A good hint for identifying the server side role is where the application data is located. In the case of sourcing, the party that plays the provider role owns the item data that is used to create the catalogues. Therefore it is straight-forward to implement the provider role as a service and make the receiver role a client.

As I learned today, UBL does not specify the actual procurement processes but merely provides the set of business documents to choose from when designing your own processes. This is nice, because it mens that UBL can be suitably applied even in the context of this little experiment.

As a first step, the provider role service will just provide a public catalogue, let's say at http://labs.nordsc.com/procurement/roles/provider/common-catalogue.

Assuming the receiver role client discovered that URL from the service document describing the provider service the following HTTP interaction could take place for catalogue provisioning:


GET /procurement/roles/provider/common-catalogue
Host: labs.nordsc.com
Accept: application/vnd.nordsc.procurement+xml

200 Ok
Content-Type: application/vnd.nordsc.procurement+xml
Cache-Control: public,must-revalidate
ETag: 1


    xmlns:cbc="urn:oasis:names:specification:ubl:schema:xsd:CommonBasicComponents-2"
    xmlns="urn:oasis:names:specification:ubl:schema:xsd:Catalogue-2"
 ">
    public-001
    2010-02-02
    
        ID0

The catalogue representation is only a stub at the moment, we'll get to the media type issues later. Of particular interest is the caching policy for catalogues because the intention is for clients to use the latest catalogue data when placing an order. Catalogues can be potentially very large and we want them to be cached by public and private caches (Cache-Control: public) . However, since the freshness of the catalogue is very important, caches are instructed to never return a cached copy without prior revalidation (Cache-Control: must-revalidate).

Apart from the issue of the media type for the catalogue, we have now designed the most simple version of catalogue provisioning. It will be interesting to explore how other sourcing interactions (for example customer initiated sourcing) can later on be added.

Service Type == Set of Possible Application States

2010-02-04T09:35:00.000+01:00

As I have said earlier, in my opinion service types are defined by the set of hypermedia semantics (media types, link relations, ..) they use. Because representations sent by services of a given type are composed of hypermedia semantics from the service type's set it can be said that the service type defines the set of possible representations.

Given that a representation returned by a server corresponds to an application state it follows that a service type defines the set of possible application states.

I like that.

RESTifying Procurement - Introduction

2010-02-03T17:53:00.000+01:00

For the purpose of this series of postings I view Procurement as a collection of business processes (e.g. sourcing, ordering) and associated business documents (e.g. catalogue, order). The goal is to design the necessary kinds of services and media types to enable the procurement processes to be executed between software components using HTTP.

A practical way to start looking for necessary services is to think about the roles played by the parties participating in the processes (in a real scenario though, legacy systems are likely to influence the actual design).

REST is a client-server architectural style, employing the principle of separation of concerns and therefore we need to differentiate between client roles and server roles. If client and server roles are not separated correctly the application state that pertains to a given collaboration might in part be located on the client which would result in unnecessary complexity. (This issue should be addressed by a separate posting but I have written up some thoughts about it earlier).

Once the kinds of services are becoming clear the media types need to be designed. On the one hand these media types must be capable of expressing the domain information expressed by the various UBL documents, on the other hand, they must provide the proper linking semantics for the business process to advance. It is important to keep in mind that resource representations primarily transfer application state and not serialized domain objects.

Once the kinds of services and the media types are designed, I'll take a look at the client programming model because one of the primary objectives of this exercise is to discuss how clients can cope with the evolvability granted to the server side by REST.

At the moment I really plan to provide an example implementation of the services to address associated development issues and make the whole example more practical. Let's see how that works out though.

This morning I have already worked on defining a manageable subset of UBL catalogue documents to support the sourcing process. Stay tuned.

RESTifying Procurement

2010-02-03T14:41:00.000+01:00

As more and more people are being attracted to applying REST in an enterprise IT context the lack of a non-trival practical example becomes increasingly apparent . Such an example would ideally cover all the aspects of RESTful integration and make visible the principles that can be applied during the individual design steps.

Ideally, the example would also help people avoid the pitfalls of unRESTful REST by discussing related REST anti patterns.

With this entry I'll start the journey to fill the void and develop an extended, practical example by RESTifying procurement.

The conceptual basis will be the Universal Business Language because it provides a customizable set of the necessary business level documents and a rather clear description of the business processes involved.

Wish me luck.

2010-02-03T13:07:00.000+01:00



GET /catalogues/public/1
Accept: application/procurement+xml

200 Ok
Content-Type: application/procurement+xml
Cache-Control: public,must-revalidate
ETag: 1234


 xmlns:ns2="urn:un:unece:uncefact:codelist:specification:IANAMIMEMediaType:2003"
 xmlns:ns3="urn:oasis:names:specification:ubl:schema:xsd:CommonExtensionComponents-2"
 xmlns:cbc="urn:oasis:names:specification:ubl:schema:xsd:CommonBasicComponents-2"
 xmlns:ns5="urn:un:unece:uncefact:codelist:specification:5639:1988"
 xmlns:ns6="urn:un:unece:uncefact:data:specification:UnqualifiedDataTypesSchemaModule:2"
 xmlns:ns7="urn:oasis:names:specification:ubl:schema:xsd:QualifiedDatatypes-2"
 xmlns="urn:oasis:names:specification:ubl:schema:xsd:Catalogue-2"
 xmlns:ns9="urn:un:unece:uncefact:codelist:specification:54217:2001"
 xmlns:ns10="urn:un:unece:uncefact:codelist:specification:66411:2001"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="urn:oasis:names:specification:ubl:schema:xsd:Catalogue-2 file:/Users/algermissen1971/Resources/UBL/os-UBL-2.0/xsd/maindoc/UBL-Catalogue-2.0.xsd">
    public-001
    2010-02-02
    
    
    
    
    
        ID0

2010-02-03T13:01:00.000+01:00

In this entry ...

(10:00 AM) - Went to the UBL committee home page and browsed UBL 2.0 specs. I have occasionally looked at UBL in the past for inspiration on UML process diagramming so I actually knew pretty much what I would find.
(10:20 AM) - Decided to start with looking at UBL catalogue documents for sourcing and loaded XSD into XML editor. To make sense of the XSD, I decided to have my editor create a document sample....
(10:30 AM) - Starring at 23,000 lines of XML...guess that was not really the right thing to do :-) So I started poking around in the example instance trying to make sense of the various nested XML levels.
(10:50 AM) - MORE COFFEE!!!
(10:55 AM) - Finally I remember having read about UBL customization guidelines for creating UBL subsets! Yep, that sounds a lot of what I need.
(11:03 AM) - Googling...

Ah:

if all optional elements in a UBL document type were instantiated, the resulting instance would be extremely verbose. For example, if a UBL Order document contained just one instance of all its possible information entities, that document would contain approximately 800,000 elements and attributes.

(From guidelines doc)

Mac OS X Productivity Apps

2010-02-02T11:23:00.000+01:00

Recently I upgraded to Snow Leopard and naturally this made me review what's in my Applications folder. Here are the more interesting products I use more or less regularly:

iWork 09
I love iWork since it came out in 2005 (?). I use Pages a lot for personal stuff because client work usually requires true Office compatibility. I always use Keynote though, never Power Point. Keynote is just great. I prefer Excel over Numbers but mostly because I know it better I guess.

Office 2008
This is a must have on a consultant's machine and I am mostly satisfied with the Mac OS version. Though Word remains as painful as ever. I love Excel - I think it is the best product Microsoft ever made.

Merlin
I am a true lover of this project management application but the occasions where I really need it are rare. I like the look and feel and the millions of export formats. It handles MS Project files nicely.

Nova Mind Platinum
The best mind mapping software ever. The Platinum version includes a script writing feature you can use to prepare presentations in a Beyond Bullet Points way.

Oxygen XML Editor
It took some research to find a nice XML editor for Mac OS X but with Oxygen I found a tool that handles everything you might want to do with XML. I mostly use it for applying XSLTs to some XML and to pretty-print one-line XML files. It is also good for generating example documents from schemas (who understands an XML schema without an example anyway?) and for schema conversions, e.g. RelaxNG to XSD.

Aqua Data Studio
Finding a really good database query tool for Mac OS X was even harder than finding an XML editor. What I wanted was a product that would connect to nearly all common databases and would provide at least the functionality of Toad for serious relational work. Aqua Data Studio is excellent and even has a relational diagramming tool and very sophisticated export and import capabilities. Very good for cycles where you need to work on relational data with a set of Unix tools.

Magic Draw Enterprise
Magic Draw is an excellent UML tools with code and database engineering support and very nice report generation facilities. I mostly use UML for documentation and I love the database reverse engineering feature to extract relational diagrams from database schemas. It is fast and has very nice look and feel for a Java-on-Mac application. But it is unfortunately not a bargain. NoMagic really does maintain and extend the product frequently so paying for the updates has always been worthwhile for me.
Magic Draw has optional plugins for SysML and DoDAF.

Eclipse Classic
Trying to stick to 'standard' Eclipse setup with no plugins except for SVN and the gorgeous vi plugin.

Omni Outliner
The Max OS X classic outliner. Always open to manage my thoughts. N'uff said.

Omni Focus
My current ToDo items application. Though I really want to look at Things.

Screenflow
My favorite tools for creating screen casts. I use this to record code walk-throughs with clients or workshops to have something to hand over afterwards.

Parallels Desktop
Virtual Machine for running Windows if I have to. I used it a couple of years ago to develop on a client's host. So it was: Remote Desktop over VPN over Parallels - worked quite nicely actually. Nowadays of course, Apple's RDT would do the job.

BBEdit
My editor if I really can't use vi or need to convert between encodings.

Quicksilver
Application launcher. Everybody must have Quicksilver, definitely!

Snippet
A little tool for managing code snippets. I often forget about using it, but its a clever thing to have.

UPDATE:

Monkey Office
After looking at a bunch of accounting software products for the Mac, this is the one I picked.

Adobe CS5
It sometimes good to be able to edit designer's work directly, for example to create translations. This saves the roundtrip of having the designer replace the texts. InDesign is an amazing piece of software.

Mondrianum
Yesterday, I learned about Mondrianum. This is a color picker that works together with Adobe's Kuler. Difficult to explain what it does - check out for yourself.

Not worth describing, but there also is EyeTV, Toast, Colloquy, Twitterific, Photoshop, Quicktime Pro, Skype.

Fond Topic Maps Memories

2010-01-30T00:38:00.000+01:00

Today I have done some Web Archeology and dug up the 2004 version of the GooseWorks.org Web site. The project was intended to support the original idea of the Topic Maps paradigm as developed by Steven R. Newcomb when two camps of ideas started to emerge from the community.

I remember doing a ridiculous amount of coding to verify and track Steve's ideas and was joined by Sam Hunting who evangelized on the various markup conferences.

Well, that's history, but I thought it might be worthwhile to put the site back online with some of the source code. Might also be that I have done so much thinking this week about steady-states, bookmarks and cool URIs that I felt I had to bring the site back to life.

In case you are wondering what inspired the name, have a look at the posting that started it all. Besides, it had the cool double-O in it that was said back then to be a guarantee for a site's success :-)

Fun with the Profile Parameter

2010-01-29T19:43:00.000+01:00

text/csv;profile="http://spec.example.org/profiles/customerdata"

http://cgi.ebay.de/Lego-Star-Wars-7656-General-Grievous-Starf-mit-BA-u-OV_W0QQitemZ260542847259QQcmdZViewItemQQptZDE_AllesfürdKind_Spielzeug_Lego?hash=item3ca990591b#ht_500wt_1059

In Fear of Sub-Requests

2010-01-27T10:38:00.000+01:00

Suppose we design a service that is to provide data about persons, including whom they are linked to. Let's say person representations look like this:


GET /persons/4/full

200 Ok
Content-Type: application/vnd.me.person+xml
Cache-Control: no-cache


  Karl Heinz
  Kieler Strasse 7,20556 Hamburg

In addition, the service provides minimal person data that looks like this (note that its time-to-live is longer):


GET /persons/4

200 Ok
Content-Type: application/vnd.me.person+xml
Cache-Control: max-age=3600


  Karl Heinz
  Kieler Strasse 7,20556 Hamburg

Now suppose we define the processing semantics of application/vnd.me.person+xml in such a way that the user agent should follow all the friend links and retrieve the referenced persons' data. Looks like rather bad design due to all the sub-requests, doesn't it?

On the other hand, nobody complains about HTML pages that link to massive amounts of images and other media.

4xx Client Error (But Not It's Fault)

2010-01-26T10:13:00.000+01:00

Erik responds to 4xx Client Error. He writes:

"i don't think it appropriate to phrase it like it's the client who necessarily did something wrong when, for example, a 404 is returned. if you follow a URI that was supposed to be persistent and you get a 404 because the server did something wrong, it's quite a stretch to argue that it's your fault. it's pretty clearly not."

I agree that my emphasis on the client doing something wrong wasn't appropriate (but an extreme position is sometimes nice :-). However, I do interpret RFC2616 to be emphasizing that the burden is on the client. While it is obviously not the fault of the client if some resource currently has no representation available (404) I would not agree that "the server did something wrong". The server is free to stop providing representations for a resource and the client just has to account for that to happen.

What is interesting about this is that the conversation between client and server is not considered broken at all because the fact that 4xx errors might occur is part of the contract (and does not indicate a broken contract). This is one of the advantages of a uniform interface with uniform status codes: it can sustain the conversation even when there is a (temporary) gap in expectations.

4xx Client Error

2010-01-25T19:51:00.000+01:00

HTTP is actually quite clear on how decoupled the server is from the client by calling all non-technical errors simply Client Errors. Dang! So simple! 404 - who broke the contract? The client! 406 - who broke the contract? The client! ...

RFC2616 uses a softer tone though:

"The 4xx class of status code is intended for cases in which the client seems to have erred."

I guess testing a RESTful system means testing the clients...

REST Doesn't Lie

2010-01-25T14:20:00.000+01:00

Have been reading about separation of concerns today and came to think that it might be interesting to shift the angle of looking at service evolution: Instead of asking what aspects of services might change it could actually be useful to ask whether there is anything that a service can guarantee not to change, no matter what future requirements it might encounter.

Shifting the point of view revealed a rest constraint that is probably so much taken for granted that it is not talked about very often:

"The only thing that is required to be static for a resource is the semantics of the mapping, since the semantics is what distinguishes one resource from another."

(Section 5.2.2.1 Resources and Resource Identifiers of Roy's dissertation)

This is a guarantee the server has to make that is actually possible to adhere to because no evolution scenario of a server can possibly force it to change that mapping. The other two things that are not subject to change are (obviously) the uniform interface and the use of descriptive message semantics.

Besides these three aspects there is nothing in a networked system that servers can guarantee to clients and REST emphasizes that in an incredibly honest way. The client may depend on the above three aspects to remain true for the entire lifetime of a service but must plan for anything else to change.

What is the take-away? Two things mostly: you are not doing REST unless your system does not adhere to the above and you should exploit the stability of the resource semantics as much as possible in your service design.

'Completing' application/atomsvc+xml - Part 2

2010-01-22T15:14:00.000+01:00

In part 1 I have described the reasoning for completing the AtomPub service document media type; in this part I'll describe some possible extensions that would make AtomPub service documents suitable for describing the initial state of any kind of Web application. Here is, again, the list of aspects that need to be addressed:

What single resources are there (single resource meaning a resource that is not inherently part of a collection of similar resources like the current temperature or the state of some system's power supply unit).

What resources are there that are collections of member resources and what capabilities do they have (for example: do they accept POST submissions?)

What resources are there that are a set of member resources and provide a generic mechanism to specify subsets (search resources)?

What resources are there that do not hold state but merely act as a gateway to some other processing component?

(The proposed extension elements below are prefixed with 'appx'.)

1. Single resources: allowing <atom:link> inside <app:workspace>
Services might expose resources that have special, individual significance to clients. Services need a way to tell the client where such a special resource can be found. For example, a service might expose the 'current state of some technical device' or 'today's featured product' and would need a way to express a link to such a resource that the client could discover. Allowing <atom:link> inside <app:workspace> solves that problem:



  
    Vehicle Control
                           title="Battery Status"/>

Clients that know the semantics of the 'battery-status' link relation are enabled to find the corresponding resource.

2. Collection capabilities: introducing an <appx:features> extension
AtomPub service documents are able to express what categories the members of a given collection have. This can be used by clients to discover collections based on their member's categories. However, in some situations services need to be able to distinguish collections by capability or abstract behavior in order to express domain semantics that go beyond standard HTTP semantics. Overloading categories for this purpose is not the right solution.
The introduction of an extension element inside <app:collection> that enables the service to describe that a collection exhibits a certain capability or abstract behavior addresses this issue:



  
    Image Conversion Service
    
       To Thumbnail Converter
       
         
       
      image/jpeg,image/gif
    
    
       Sharpener
       
         
       
       image/jpeg,image/gif

The above service document describes two collections that each exhibit a special behavior that is significant to the client. Based on the desired conversion to be performed on a submitted image it can choose the appropriate collection. This is a scenario where the category or media type of the submitted representation is orthogonal to the capability of the collection.
(This extension element is based on the original draft-snell-atompub-feature Internet draft by James Snell).

3. Search: introducing an <appx:search> extension
Despite search being an almost ubiquitous requirement AtomPub does not currently support the description of search capabilities of a service. Support for search can be added to AtomPub service documents by introducing an extension element that enables the service to describe accepted parameters, format and media type of the search resource.



  
    Product Search

Clients can process this form and construct the query string for the GET request. There are several other aspects defined by the OpenSearch specification that would make a lot of sense to add here.

4. Processors: introducing <appx:processor> extension
Occasionally services expose resources that merely act as a gateway to some other data submission processing component. Such resources do not exhibit the POST-as-append semantic of AtomPub collections and therefore using <app:collection> to describe such resources is not appropriate. Instead, an extension element is introduced to describe processor resource capabilities:



  
    Customer Relationship Services
    
       Catalogue 'Punch Out' Fax
       
         
       
       application/vnd.foo.procurement-catalogue+xml

The above processor description tells clients that there is a data submission accepting process at /channels/punch-out that features the abstract behavior 'punch-out' and accepts application/vnd.foo.procurement-catalogue+xml request bodies.

Note: I am not entirely convinced of the necessity of <appx:processor> because any data processing resource could always create a resource upon submission that represents the processing request. That way <app:collection> could be used instead. The members of the collection could simply be HTML documents that describe the processing request. It would also be possible to just not implement GET for the AtomPub collection resource and return 200 Ok (without Location header) for the data submission. This is not fully in sync with AtomPub's description of the operations, but an HTTP client must be prepared to handle any return code anyway.

Conclusion
I doubt that the above will be deemed relevant to the Atom Publishing Protocol, but I think it would be very useful to have a general service description document format. Especially so in enterprise IT contexts, where the number of services is high and a uniform means of service description simplifies matters considerably. In fact, I have needed something like the above in every project so far and used almost all of the extensions above in my latest one.

The rationale really is that there is in my opinion no possible requirement for service descriptions that cannot be addressed by the above generic format. Since application/atomsvc+xml already covers most of it, why not combine it with the extensions to form a new media type?

When I find the time I'll probably submit this as a request to register application/service+xml.

Why if(status == 200) Is Not Enough

2010-01-22T09:58:00.000+01:00

In another Developer Works article I read yesterday, Bruce Sun provides a code snippet that acts as an HTTP client to a service

REST Design Mistake with Apache Wink

2010-01-21T23:22:00.000+01:00

Just came across an article on service implementation with Apache Wink by Vishnu Vettrivel. While it is primarily covering the service implementation aspects, it also has a section on RESTful design that too easily creates a wrong understanding of how a RESTful service would have to be designed.

Given that REST has entered the hype cycle it happens too often now, that actually unRESTful design is promoted. It is probably time to spend more energy on preventing that REST will be doomed before enough adopters have indeed realized and, more importantly, experienced the benefits. REST done badly might actually cause more harm than WS-* done well, so lets be wary!

What is wrong about the article? Three things mostly:

Likely violation of the hypermedia constraint: the article proposes a URI structure but fails to make clear that this knowledge must not leak to the client developer but that the client should discover the appropriate links or forms at runtime. Coupling on the URI structure must be avoided to allow the server to control (and change) its own URI space without the risk of breaking clients.

Misplaced authentication information: the proposed design places user name and password as parameters into the URI while they should be send to the server using standard HTTP authentication mechanisms. This can negatively impact caching since caches understand the special meaning of the Authenticate header. In the worst case, a public cache might keep a representation that should only be visible to an authenticated client.

Limited visibility: the article proposes the use of application/json but fails to make clear that this causes coupling on the out-of-band knowledge of the actual JSON structure being used.

I am aware that RESTful design is not at the heart of this article, but we should make sure that whoever reads what we write does not walk away with a wrong impression of what a RESTful design is.

'Completing' application/atomsvc+xml - Part 1

2010-01-21T18:37:00.000+01:00

The Atom Publishing Protocol specifies the media type application/atomsvc+xml for service description documents. At first glance the things that Atom Publishing Protocol service documents can describe seem quite specific. However, from a more general point of view, AtomPub service documents are really just describing what collections of resources are being made available by a given service and which are the capabilities they have. AtomPub service documents are describing something about a service that applies to any kind of service, not only AtomPub services.

We might as well turn the line of thought around and ask: "What are the aspects that make sense to be described for any kind of service (not only AtomPub services)?"

Now, it only makes sense to describe something about a particular service that is specific to that service and therefore needs to be told to the client at runtime. The only thing in a RESTful service that is specific to a particular service is through witch resources it provides access to the application data and associated processes it manages and how those resources reference each other to constitute the state machine that the client can follow. Anything else is design time information described by hypermedia specifications that apply to a broader context than a single service.

A second aspect that is important in this context is that it is (of course) not desirable to describe the whole set of resources and their relationships because this would lead to unnecessary coupling. Clients just need to be put into the initial application state by downloading the service description document. The description of this initial state is what service documents are necessary for.

The original question can be rephrased to: "What are the aspects that constitute the initial state of a RESTful application without introducing any particular domain semantics?".

In my opinion, there are four aspects that apply to any RESTful application and can therefore be considered general:

What single resources are there (single resource meaning a resource that is not inherently part of a collection of similar resources like the current temperature or the state of some system's power supply unit).

What resources are there that are collections of member resources and what capabilities do they have (for example: do they accept POST submissions?)

What resources are there that are a set of member resources and provide a generic mechanism to specify subsets (search resources)?

What resources are there that do not hold state but merely act as a gateway to some other processing component?

[Side note: you might want to look at Mark Baker's Abstract Model for HTTP Resource State; it is somewhat related.]

With the premise that the items above are general enough to apply to any kind of Web application (service) wouldn't it make sense to have a service document format that is capable of expressing information along the lines of those items?

I think it does and I also think (and this is the motivation for this posting) that application/atomsvc+xml service documents already cover a substantial part. Which might, after all, be the reason why AtomPub seems such a nice fit in many scenarios.

In part 2 of this posting I will present a couple of extensions to application/atomsvc+xml that make this service document media type 'complete' in the sense of being able to express items 1-4 above. This will cover:

Allowing <atom:link> in <app:workspace> elements to express 'single resources'

A <feature> extension to express abstract capability or behavior of a collection

A <search> element to express search resources (much like OpenSearch does)

A <processor> element to express resources that accept data submission but are not a collection in the AtomPub sense.

I am pretty convinced that such a format would be able to represent the initial state of any RESTful application and since AtomPub provides all the plumbing already it is probably worth the effort.

Read part 2.

The Protocol Hammer

2010-01-21T02:10:00.000+01:00

It's not news, but Anne's statement just popped out when I scanned through the archives:

"If I knew what I know now back in 2000, I would have pushed for a RESTful registry with free-form search, and such a beast would have been a lot more valuable than UDDI. But we had this really spiffy, state-of-the-art protocol hammer called SOAP, and we saw everything as a nail."

Reminds me that REST officially turns 10 this year!

Test

2010-01-20T14:04:00.000+01:00

Das ist ein Bild so wie es sein solll

Und das ist gut

Service Type Specifications II

2010-01-08T13:08:00.001+01:00

UPDATE: My thinking around this issue has evolved.

Wow, this question was much harder to get straight than I expected:

What constitutes the specification of a service type?

First things first, though! In order to provide a principled answer to that question, another one needs to be answered before:

What is the purpose of a service type specification?

Purpose of Service Type Specifications

1. Service type specifications provide the information client developers need to implement a client for services of the described type. If you hand the service type specification to a developer she should be able to know exactly what to do and what to reasonably expect from any instance of that service type. There should be no further knowledge required (except for following any included references, for example, to media type specifications, of course).

2. Service type specifications provide the information necessary for implementing instances of the specified service type. There should not be any further information required except for implementation specific details behind the service boundary of course.

3. Service type specifications provide the information service owners and maintainers need in order to understand in which way the server can evolve without breaking clients. This is redundant with 2. but mentioning it explicitly emphasizes where exactly the contract is established between client and server owners. Anything that is not specified in the service type specification or referenced material is not part of the contract and constitutes no obligation by either party.

4. Service type specifications enable service discovery by type. Clients that wish to interact with a certain kind of service can use the information provided by the service type specification to detect when they see a service that is of the desired type. The necessary information should be available as a response to the published service URI either by analyzing the set of initial transitions (goals) provided by the service or by looking at the service document's media type (see below).

Having laid out the purposes of a service type specification, we can now make principled decisions about what should be part of a service type specification.

Elements of Service Type Specifications

Here is the short answer (rationale follows below):

Service Type Specifications define the set of hypermedia specifications (media types, link relations, etc.) used by the service and information about the minimal initial set of available transitions (goals).

The latter can optionally be expressed as a media type, too (a service type specific service document type), which simplifies the definition to 'a set of media types' but leads to the creation of a new media type for a given kind of service.

The essential aspect of the above definition is that the client needs to know what the media types are it needs to understand in order to interact with the service.

So, why is that?

The ideal situation would be to say nothing about the service type at all, just agree on a set of media types that make sense to be understood in general and implement all or any of them in the clients as desired. The problem with this approach is that it does not address purpose 1. above; a client developer would not have any clue what a service is doing or how to interact with it. There would not be any notion of a service type at all; just individual hypermedia semantics (media types, link relations etc.).

But even if the client developer was provided with some means of a service type description (in the form of a dedicated service media type or a set of initial transitions) - see purpose 4. above - there would still be no way for the client developer to have any clue what can be done with the service beyond the initial transitions. Knowing the set of media types provides that clue.

The issue of guiding the service developer (purpose 2.) is addressed because the service type tells the service developer exactly what media types etc. are available to him to solve the given implementation task.

Purpose 3. above is addressed by the fact that it is not possible to remove a hypermedia specification from the specified set without incompatibly changing the semantics of the service type. Service owners are therefore free to evolve the service by adding hypermedia specifications (or extending extensible ones) but may not remove any.

The purpose of discovery (4. above) can be addressed by using a generic service description media type such as Atom Service documents (application/atomsrv+xml) and describing a minimal set of available resources. For example, an ITIL-conforming help desk service type could be specified as providing at least three collections (identified by categories) containing and accepting submissions of Incidents, Problems and Change Requests. Clients (including service registry crawlers) would know they come across an instance of that 'ITIL helpdesk service' when they see an Atom Service with the specified collections.

Alternatively, a new service document media type could be minted (e.g. application/helpdesksrv+xml) to identify the service type. The format of that type would be defined to provide links to the necessary resources. This leads to simpler discovery/registry mechanism but also might cause explosion of media types.

Using a combination of both might actually be best, such as application/atomsrv+xml;profile=helpdesk.

Service Type Specifications

2009-12-07T11:58:00.001+01:00

Over the last weeks I've been busy with finding a suitable approach towards REST Web services descriptions. Not quite there yet, but close! Here is my list of essential aspects a service type specification must address:

Overall purpose of the service (aka the realized business process role)

What client goals does the service enable (aka protocol operations)

When can the client expect certain goals to be available (aka partial ordering of application states)

What does the client have to do to 'execute' a certain goal (aka hypermedia semantics)

Separation of Concerns and Replication

2009-11-27T11:29:00.002+01:00

In an integration scenario that involves the replication of items from one system (the master) to another (the slave) the direction of the communication is an interesting issue.

First, there is the question, whether the initial creation of a replicated item in the slave should be performed by a request from the master to the slave or by a request from the slave to the master. The former will often be chosen when there is a business rule involved that should trigger the replication (e.g. replicate all employees in the slave system whose contract ends in 6 months). The latter is more likely to be used in a scenario where the slave replicates all new items in the master, in other words, where the slave keeps the overall set of items in sync.

Second there is the question how updates of the items in the master are made available in the slave. This is the classic distinction between polling and publish/subscribe: Should the master notify the client of updates or should the slave poll for updates?

In addition, there is sometimes a desire to notify the master about changes the slave makes to the replicated items. The intention of this scenario is not the bidirectional synchronization of master and slave but reflects the master's interest in any changes the slave makes.

What is the guiding principle to make an informed design decision regarding the direction of communication in such a replication scenario? The answer is separation of concerns with the goal of simplicity and avoiding unnecessary coupling. This leads to the question which of the systems should for which communication play the server role and which one should play the client role?

Every communication is initiated to achieve some goal and the correct separation of concerns puts the component that acts to achieve the goal into the client role and the other component into the server role.

Applied to the above scenarios this means that for initial replication and subsequent updates the slave system should act as a client and the master should act as a server. All the necessary resources to be provided by the sever (e.g. providing list of all items that match a business rule) can be derived.

Applied to the scenario where the master is interested in updates from the slave it means that the master should act as the client because it is the master that has the goal (the slave does not care at all). This leads to certain resources that need to be exposed by the slave.

I have seen at least three production systems that had this issue and all of them did it wrong and would have greatly benefited in terms of simplicity and loose coupling from following the above principled design path.

Related is Roy's post Paper tigers and hidden dragons.

History of Fragment Identifiers

2009-11-26T01:01:00.002+01:00

Everything you ever wanted to know about fragment identifiers - an in-depth coverage by Roy back in 2002.

300 and 406 Response Bodies

2009-11-25T21:44:00.002+01:00

Reading through some older postings, I came across this by Roy:

That is why we have a WebDAV working group.  Both the 300 and 406 response bodies were left unspecified because the intention was that they be specified by a group that actually had time to study the problem in detail and come up with a [hopefully] better solution than some off-the-cuff invention of mine.  It was one of the WebDAV to-do items, last time I checked.

Has this ever been done? Maybe it is time to create some standard 300 and 406 response media type?

UPDATE: Umm, should have searched a bit more I guess: RFC2295. What is left is to tell clients to look for Alternates headers in 300 and 406 responses.

ESB Decoupling Capabilities and the Elegance of REST

2009-11-17T00:25:00.001+01:00

In his presentation "The Role of the ESB" Mark Richards makes the point that the most significant capability of an ESB is that it enables the separation of "Business Services" from "Service Implementations" (at 00:11:25). "Business Services", he says, "are the ones that make sense to the business" while "the implementation service [...] is the one we actually code". As an example he contrasts "PlaceTrade" and "saveTradeOrder()" and talks about how PlaceTrade is much more stable than saveTradeOrder() because the former is a business semantic and not an implementation detail.

While it is truly astonishing what amount of tooling and complexity the slide conveys in order to provide a business service 'PlaceTrade' it is something else that struck me as being really interesting:

It is obvious that, in order to provide a 'PlaceTrade' service, there needs to be implementation details that should not be conveyed to the user of 'PlaceTrade'. (According to Richards, SOA wonderfully solves that problem by using an ESB that translates the business service into an implementation service call. Can't help it...enterprisey comes to mind...)

REST takes a different approach: in a RESTful system, the implementation details that correspond to the translation of 'PlaceTrade' to 'saveTradeOrder()' manifest themselves in the form of the resource model of a service (the actual resources the service chooses to make available). This is a direct consequence of realizing business functionality through a uniform interface.

Interestingly, in a RESTful system, clients are inherently decoupled from the services' resource models - the contract between client and server is expressed solely through hypermedia specifications, independent of any actual service implementation.

In other words: SOA makes use of ESBs to achieve a decoupling that is already inherent in REST per design.


POST /trades
Content-Type: application/trade-order

<trade-order>
....
</trade-order>

Service Descriptions and Legal Contracts

2009-11-05T14:19:00.001+01:00

The intermediate result of the thoughts triggered by the recent interchange on the SOA list are here. Comments welcome!

Formal Contracts

2009-11-03T13:22:00.001+01:00

When clients in a RESTful system are not ultimately driven by a human then the expectations they make about the resource model of the services and the available representation formats will inevitably be manifested in the client side code to some extend.

While the client developer can go along way in limiting the assumptions made about the server at some point the client code just needs to explicitly assume that there is for example a 'place order' state traversal (aka 'link' or 'form') when interacting with a service were you can order stuff. The client can plan for various ways in which the server could express the link or form but ultimately the client code will contain some form of orderProcessorUri = clientLib.lookupUri(...expression to find order processor URI...).

If the traversal option cannot be found the client will break without a possibility to automatically react. This is where a human user would change the online store or call the hotline.

Steve Jones is concerned about the development process related- and legal implications of the lack of a formal means for documenting such implicit contracts created between client and server once the client manifests its assumptions in its own code or configuration. I think he is making some valid (and too often ignored) points.

I have been working on related questions during the past weeks and currently see the following issues:

While the freedom that is usually (deliberately) given to servers in Internet specifications improves evolvability it is not a sufficient basis for establishing legal contracts between service consumers and service providers. If you make a serious investment into implementing a client to some service then RFC 2119's SHOULD is probably not the best contractual basis.
If a service makes use of a variety of media types, extensions, link relations, categories, etc. then how would one express this in a formal way?
If a client is dependent on the presence of some Atom extension in Atom entry documents for example, how could the service provider state that they are committed to providing that extension?

It is important though to emphasize the general goal of maintaining the flexibility induced by REST's constraints and at the same time to make explicit and to formalize the inevitable coupling that occurs when non-human clients interact with services.

This is important not only for development-process and legal matters but also for IT governance and Service Management issues and critical for the adoption of REST inside the enterprise.

Hidden Contracts and Governance

2009-10-27T20:24:00.001+01:00

In a RESTful system the contract between client and server is expressed solely in the form of hypermedia specifications[1]. On the Web this works fine, because there is usually always a human being involved that can act as the ultimate arbiter of semantics when things need to be decided that happen not to be covered by the involved hypermedia specifications.

Examples:

1. When I want to buy a book, what hypermedia is telling me that www.amazon.de is a place to go?

2. When the Amazon site is not reachable, how do I find out where else I could buy a book?

3.How do I know what 'buying a book' means and what the basic sequence of operations is that need to be executed in order to achieve the overall goal?

4. How do I figure out, what the new appropriate sequence of operations is should Amazon significantly change its understanding of the flow of a book purchase?

All of the above are ultimately solved by a human being in the case of the Web and usually don't cause any trouble.

3. is particularly interesting because the fact that no hypermedia specification even touches that question reveals that there is a very high level, common sense contract in operation between human clients and some Web applications.

4. is also interesting because economic sense will usually keep e.g. a book seller from changing the flow of operations in a way that causes the potential buyer to not being able to buy anymore.

How does all this relate to IT governance?

When one applies REST (HTTP that is in practice) in an enterprise IT context, these hidden contracts matter because they either need to be covered by hypermedia specifications (e.g. for the discovery/equivalence examples 1. and 2.) or at least need to be made explicit so they become manageable in an IT governance sense.

Example 3. relates to the general question of how types of services (e.g. 'procurement seller role' or 'repository') can be expressed to provide guidance to client developers regarding the intended sequence of operations.

Example 4 relates to the question of how change can be managed not of the pertaining hypermedia specifications (this is easy!) but of their composition (which is what effectively forms the service type).

Besides being able to frame the problem and ask the questions I do not yet have definitive answers - comments and thoughts are very welcome.

lass="Apple-style-span" style="font-size:medium;">[1] The term 'hypermedia specification' is meant as a general expression for specifications of media types, media type extensions, link relations and 'little' things like category definitions (used for example by Atom and NewsML2).

(Also posted in the XING REST Group)

Taylor, Medvidović, Dashofy. 2010. Software Architecure

2009-10-26T15:18:00.001+01:00

The book just arrived and I think it is one of those worth an in-depth study!

I like the caution note on page 421:

"Numerous Web sites and books purport to characterize or exemplify REST principles. The reader should be very cautious, as many sources misrepresent or mischaracterize REST."

Sadly so..

Product Modeling at W3C

2009-10-10T19:03:00.001+02:00

This seems sort of interesting and probably has some potential to be used in Web architecture-based procurement collaborations. Is there any serious industry involvement in this?

I wonder if OWL and friends will eventually obfuscate the real goal; wouldn't a decent media type be a better way to approach this?

Please use 'Hypermedia Constraint'!!

2009-10-06T14:55:00.001+02:00

The hit count of this should go to zero! This should skyrocket instead.

Formal Hypermedia Contexts

2009-10-01T08:35:00.001+02:00

Formally, a hypermedia context can be understood as a conjunction of predicates that evaluates to true or false for a given URI. The predicates can be expressed as one of the following:

unary link, e.g service(x)

binary relationship, e.g. edit-media(u1,x)

pattern match on representation of x, e.g. r-match( xpathExpr , x)

pattern match on occurrance of x in representation, e.g match( xpathExpr , x)

Logical sentences build from such predicates can be evaluated against the current state of the client to either test if a given URI matches the denoted context or to find all URIs known to the client that match the predicate conjunction.

The next step will be to express the operations that are currently stated in the form of prose in hypermedia specifications in terms of the above predicate conjunctions and HTTP operations. For example could AtomPub's 'create entry' operation be expressed as the combination of the predicate conjunction match( '/service/workspace/collection@href',x) and POST. Where the match predicate evaluates to true for any URI x that is found as the value of the XPath expression.

This needs more precision of course, but it's a start.

RFC 5545

2009-09-30T19:48:00.001+02:00

iCalendar is now an IETF Proposed Standards Protocol (superseedes RFC 2445). Does anyone know whether this is just fixing things or whether there are additional semantics that increase the expressiveness?

Hypermedia Context

2009-09-30T07:52:00.001+02:00

Descriptions of REST interfaces often use the notion of types when talking about the exposed resources. The Atom Publishing Protocol for example includes a section that provides a resource classification. Such classifications are not against the RESTfulness of an interface as long as the classes are expressed in terms of the knowledge a client has acquired about a resource. The Atom Publishing Protocol for example states that "a Resource whose IRI is listed in a Collection is called a Member Resource". This is a valid classification from the point of view of its RESTfulness.

In order to emphasize that such classification can only be done in terms of the acquired application state of the client I suggest the term "hypermedia context" instead of "classification". REST interface descriptions could then for example state: "The hypermedia context of a resource is called Member if its URI is listed in a collection". There is probably also value in distinguishing the client-acquired knowledge about a resource from the context used in the specification by calling the former acquired hypermedia context or just hypermedia context and the latter named hypermedia context.

I can also see possibilities to formalize hypermedia contexts along the lines of predicates, for example as an AND combination of known links, appearance of links in received hypermedia and (XPath-)matches on the representations of resources.

But I'll address that in a follow up.

Location Header Semantics

2009-09-29T14:16:00.001+02:00

Depending on the response's HTTP status the Location header has slightly different impact on the client's application state. The following table summarizes what I extracted from the HTTP spec.

Status	Semantic	Comment
201 Created	URI of created resource	Client is notified that a new resource has been created an can dereference the URI to see what has been created, e.g. to verify its expectations. ETag can be used to provide current ETag of created resource
204 No Content	URI of created resource(?)	Clarification pending
300 Multiple Choices	URI of server's preferred variant
301 Moved Permanently	New permanent URI
302 Found	Temporay URI to use for requests to the intended resource
303 See Other	Response to the POST request is found at another URI	If client follows redirect and if the client already knows the target resource the 303 works as a notification that the previous POST has changed the target resource.
305 Use Proxy	URI of proxy to use for the request.
307 Moved Temporarily	Temporary URI to use for requests to this resource.

201 and 303 provide the grounds for interesting generic client behavior. About the nature of 305 I am not sure yet but the server's ability to redirect a client to an intermediary smells interesting.

REST Interface Implementation Design

2009-09-18T18:19:00.001+02:00

It occurred to me today that there is a subtle design artifact that sits in between the specification of a REST API and its actual implementation. Let me try to explain what I mean:

The specification of a REST API consists of the media types, extensions and link semantics that a potential client must understand to be able to meaningfully interact with the API. The combination of Atom and the Atom Publishing Protocol is a good example of this.

The implementation of a REST API maps the 'back end layer' to a number of resources that provide access to the 'back end layer' in terms of the semantics of the REST API.

This looks like the expected differentiation between interface and implementation, much like you implement an OO interface with a class.

Interestingly, in the case of REST there is a third design artifact that resides somewhere in the middle between interface and implementation: the design decision of what resource are actually exposed and which of the interface semantics are actually used in which circumstances. It could be argued that this decision is part of the implementation, but by distinguishing the two it becomes clearer that there are many potential ways how different systems can manifest a given REST interface.

A class that implements an OO interface will always just implement the methods that constitute the interface, no decisions besides the actual implementation, no surprises. When implementing an Atom Publishing Protocol service, there is an unlimited number of ways to organize feeds and add links to feed or entry documents. One can even build two entirely different APP facades around a given back end but still use the same code to map back end resources (general sense) to feed and entry documents.

When REST API specifications define what resources a client will find on the server side and what type they have then this is an indication that the specification (wrongly) includes the server-side design artifact described above. Such specification elements must be removed from the API to make it RESTful.

Parameters are Link Relation Specific

2009-09-17T10:41:00.001+02:00

I used to view REST APIs to be constituted by media types, extensions, link relations and parameters (query-, form- and URI template parameters). Reading through Roy's checklist again today I realized that he suggests a superior approach: By making the parameters link relation specific the problem that parameters are usually not bound to a name space goes away and there is one less class of design artifacts.

Applying the Web to Enterprise IT

Generic Media Types Potentially Leak The Model

REST and the Enterprise

Agency Boundary

Interesting Changes *Must* Surface

Artifact Ownership or When Not to Use REST

Generic vs. Specific Media Types and Evolution

Need For User Involvement vs. General Applicability of Media Types

Should we maybe have a TEST method?

NetBeans 6.9.1, GlassFish 3.0.1 and Jersey 1.5

Warning:

Install Netbeans 6.9.1

Remove GlassFish JAX-RS libraries

Download Jersey 1.5 GlashFish-ready Package

Download Jersey 1.5 View Client Package

Create a New NetBeans Library for the Jersey JARs

Create Example Application

Notes

Spotted Alternates Header in the Wild

GET /stock-quote/foo vs. getStockQuote("foo")

PUT and Content-Location

How much machine do you need?

Thoughts on Media Types

Amazingly Close

Serendipitous Reuse vs. Reusable Services

IRC Conversation on User, User Agent, Media Types, Application etc.

Jersey Client Side: The ViewResource Class

A RESTful Client Side Framework for Jersey

Steady-State

Why 'Action Resources' are a REST Anti Pattern

DELETE and 201 Created

The Farmer in the Dell

Classifying the CouchDB API

The HTTP-API Family

Service Types Revisited

Markus Karg on the Hypermedia Constraint

Markus Karg on the Hypermedia Constraint

Three Aspects of Steady-States

RESTifying Procurement - Service Running

RESTifying Procurement - The Media Type

More Comments on Jersey

Jersey Going Astray?

RESTifying Procurement – Sourcing Part 2

REST Web services with Yaws Yapps

Setting up Erlang, Yaws and CouchDB

RESTifying Procurement - Sourcing Part 1

Service Type == Set of Possible Application States

RESTifying Procurement - Introduction

RESTifying Procurement

Mac OS X Productivity Apps

Fond Topic Maps Memories

Fun with the Profile Parameter

In Fear of Sub-Requests

4xx Client Error (But Not It's Fault)

4xx Client Error

REST Doesn't Lie

'Completing' application/atomsvc+xml - Part 2

Why if(status == 200) Is Not Enough

REST Design Mistake with Apache Wink

'Completing' application/atomsvc+xml - Part 1

The Protocol Hammer

Test

Service Type Specifications II

Service Type Specifications

Separation of Concerns and Replication

History of Fragment Identifiers

300 and 406 Response Bodies

ESB Decoupling Capabilities and the Elegance of REST

Service Descriptions and Legal Contracts

Formal Contracts

Hidden Contracts and Governance

Taylor, Medvidović, Dashofy. 2010. Software Architecure

Product Modeling at W3C

Please use 'Hypermedia Constraint'!!

Formal Hypermedia Contexts

RFC 5545

Hypermedia Context

Location Header Semantics

REST Interface Implementation Design

Parameters are Link Relation Specific

Interesting Changes Must Surface