I don't think I have confused HATEOAS with "connectedness", I have a reasonable understanding of resource state, application state and HATEOAS.

See http://www.stucharlton.com/blog/archives/000141.html and this discussion, http://tech.dir.groups.yahoo.com/group/rest-discuss/message/12497, which I see you have contributed to.

I think this final point made by Roy is pretty applicable here:

A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API). From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations or implied by the user’s manipulation of those representations. The transitions may be determined (or limited by) the client’s knowledge of media types and resource communication mechanisms, both of which may be improved on-the-fly (e.g., code-on-demand). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]

My point is you can't adhere to the HATEOAS constraint, without providing links between resources so the server can guide the client to other application states.

I agree that forms are a better example for explaining state transitions, especially since resource state is likely to be changed by providing forms (as will the User Agent, as Roy mentions); or equivalent form templates/prototypes for other media types.
But to me providing forms or equivalents for other media types is a different and easier problem to solve.

I'm very interested in how to evolve existing systems; I want to leverage the vast islands of information that already exist.

I just want to do it in an elegant way from a code and a web perspective, including the fact that uri's shouldn't change.

URITemplateAspects; LinkAwareModels and LinkTemplateProcessor

In a standard, layered design, you would have these components (see Image):

1. Code-On-Demand
2. View
3. Controller
4-6. Contains the Proposed Abstractions
7. Model
8. Persistent Store - Db; file; jcr; webdav
9. Proxies

Here are some options I have for adding links around some existing system (there are likely more)

Due to the time constraints, the Pros, will be sparse.

Option 1: Add Links via Code-On-Demand.

Add links to my html page via javascript.
I could take my existing html representation and use dhtml to add links.

Cons:
* code-on-demand not available for all user agents or may be disabled
* Proxies can't cache the entire representation
* leak of possible application states to clients that don't need to know
* Detailed knowledge of resource and subresources required to provide "link rich" representations
* Alternative media types may require alternative languages or structure to perform add the links. ie. use of xsl for xml; javascript with svg.

Option 2: Add Links via Views

Cons:
* We all know this is bad
* Detailed knowledge of resource and subresources required to provide "link rich" representations
* different views/media types require the same logic and detailed knowledge of the model
* different views/media types may require some application state knowledge. an admin user may have other links than anon.

Option 3: Add Links via Controllers

Pros:
* Can handle the inclusion/exclusion of links depending on the current application state, regardless of the media type to respond with

Cons:
* Detailed knowledge of resource and subresources required to provide "link rich" representations

Option 4: Add Links via Models

Pros:
* Detailed knowledge of the model and related models belong in the Model

Cons:
* Detailed knowledge of resource links aren't the domain of the models
* Not all possible related models are likely to be represented; to balance performance with types of usage, often some relationships are not modelled, so as to not eager load, very large graphs etc.
* Links are very static, The "next available" application state, still needs to be determined in another layer, like Controller

Option 5: Add Links via Persistent Store

Store partial links in the db

Pros:
* Db driven, easy to update
* Tables have relationships
* easy adoption path
* easy to link to external resources, or resources outside your db schema
* easy to share links at db level, if several systems integrate at that level
* Good for sparse relationships.

Cons:
* Update to change a link, will not play nice with intermediaries nor "cool uris, that don't change"
* Not great for consistent dense relationships, you probably want to model that differently. So with a Parent Child relationship, you wouldn't want something in addition to existing foreign key relations, to redundantly specify that to get to the children of a parent you have a list of /child/xyz links. but by having that specified in the URITemplateAspect then you are managing that once, close to the data.
* Relationships are not always defined in Foreign Keys
* Links are very static, The "next available" application state, still needs to be determined in another layer, like Controller
* Only a partial solution for more dynamic aspect of determining what application states make sense, still needs to be done in another layer.
* May be invasive, links stored in columns or other tables, no good if you've got a locked down schema.

Thoughts

Models shouldn't have relationships to hardcoded resources, like "/resource/50", but they could have a semantic relationship to "/resource/{id}" or "/resource/{id};role={role}" under certain conditions.

There is a place for such LinkAwareModels (Component no. 5), as long as the links are isolated and only contain semantic relationships, they are not urls.

The Links from Models to Resources are isolated and encapsulated within its own abstraction; which gives us component no. 6 URITemplateAspects (maybe just simplify this to LinkTemplateAspect).

Possible "Application States" are dependent on the request, even though a request must be STATELESS, possible "Application States" may still be determined by the security access; restrictions of media-types or client, which should all be part of the request. If you do stuff with user roles, then make sure you encode within your url.

Component no. 4 is our LinkTemplateProcessor(s) and its responsibility is to evaluate URITemplates within the LinkAwareModels/entities and to determine appropriate state transitions based on the request.

This leads us to the final Option ( well, for this post )

Option 6: Semantic relationships of a "Model to resources" definied via URITemplateAspect and a LinkTemplateProcessor to evaluate the relationships (URITemplates) and sub-entities to determine other "application states"

Pros:
* Detailed knowledge of the model and related models belong in the Model
* Detailed knowledge of the model to resource of interest is a concern isolated in the URITemplateAspects
* Any possible relationships to other resources can be defined
* Links are very dynamic. The possible state transitions can be determined by the LinkTemplateProcessor, but it can do so decoupled from the detailed relationships of the model to other resources.

The semantic relationships that are defined in the LinkAwareModels, can include uri and other attributes, like rel. The LinkTemplateProcessor can evaluate a lot of the current application state (but not all as the User Agent may have representations from other servers or representations that have been modified by Code-On-Demand) in combination with the semantic relationship to determine the ultimate value of the link to use (relative preferred, of course) and it can determine if that link is a valid state transition. Only if you are an admin do you see this link; or the link retains the current media-type used for this request; so a request for /country/AU.html includes links to cities.html, not just cities. The default negotiation might be something if that is not specified.

* The Processor can operate on objects, prior to rendering to a particular media-type, so if you have a single object model that can render to multiple media-types, you could have less code

Conclusion

What do you call, this, maybe Rich Interconnected Model View Resource RIMVR.
Rich in relation to having deep links within a resource representation and from the applicability of those links. That actual uri templates are evaluated late and with some context of the request and application state.

The use of these abstractions, in no way make an application RESTful, that is up to the developers, but hopefully by following some abstractions, like these we will get some clean code and some better understanding of RESTful APIs.

I have some old code, to show this working.
One day I'll get a chance to clarify terms and simplify the code

Tweet This Post

One of the best aspects of RESTful orientated services is getting your resources connected, possibly even achieving HATEOAS. Making that jump with with legacy object models or data can be tricky, depending on how intrusive you want to be.

There is such a large amount of legacy data in various forms, usually with minimal links/relations that could be leveraged.

So it is a worthy challenge to try and do it in the least intrusive manner possible.

This is one approach that is reasonably decoupled from an existing legacy model.

There are three main parts to this technique:

1. Enhance your model objects so they have a placeholder/collection so you can accumulate links
2. Annotate your model at interesting points with link templates, around attributes or methods or classes
3. Use a custom ContainerResponseFilter class of jersey to scan for annotations of interest, then evaluate your link templates (at run time), using the data of your models and any request based information you may require.

I chose to enhance my model classes using AspectJ at compile time.

Download the source

mvn glassfish:run
./curl_test #will hit two urls

and you should see some results, with links like this:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<country xmlns:ns2="http://www.w3.org/1999/xlink">
  <links>
    <link rel="" ns2:type="locator"
        ns2:href="http://localhost:8080/hateoas4legacy/rs/cities/Brisbane"/>
    <link rel=""
        ns2:type="locator" ns2:href="http://localhost:8080/hateoas4legacy/rs/cities/Sydney"/>
    <link rel=""
        ns2:type="locator" ns2:href="http://localhost:8080/hateoas4legacy/rs/cities/Melbourne"/>
    <link rel=""
        ns2:type="locator" ns2:href="http://localhost:8080/hateoas4legacy/rs/countries/AU"/>
  </links>
  <cities>
    <city>
      <links/>
      <name>Brisbane</name>
    </city>
    <city>
        <links/>
        <name>Sydney</name>
    </city>
    <city>
      <links/>
      <name>Melbourne</name>
    </city>
  </cities>
  <isoCode>AU</isoCode>
  <name>Australia</name>
</country>

All countries Results

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<countries>
<country xmlns:ns2="http://www.w3.org/1999/xlink">
  <links>
    <link rel=""
        ns2:type="locator" ns2:href="http://localhost:8080/hateoas4legacy/rs/countries/AU"/>
  </links>
  <cities/>
  <isoCode>AU</isoCode>
  <name>Australia</name>
</country>
<country xmlns:ns2="http://www.w3.org/1999/xlink">
  <links>
    <link rel=""
        ns2:type="locator" ns2:href="http://localhost:8080/hateoas4legacy/rs/countries/USA"/>
  </links>
  <cities/>
  <isoCode>USA</isoCode>
  <name>United States of America</name>
</country>
</countries>

It is very crufty, but does work in the simple cases. I haven't had a chance to demonstrate all of the features yet.

I'm likely to further flesh out this example as it is interesting in other RESTful related ways.

Tweet This Post

Anyway I liked the Circuit Breaker pattern, as I've seen a number of apps with highly coupled systems that barf once latency increases in the back ends then you get "Cascading Failures" and the webapp's request handler thread pool is exhausted and they all wait their mandatory 30 seconds for a response. Eventually the app either dies, or gets taken out of a load balance farm causing a "Chain Reaction".

Here is an Experimental Circuit Breaker Implementation (circuit_breaker.tgz) , NOT built or tested FOR PRODUCTION use in anyway. There is explicity no thread safety no the actual CircuitBreakerSimple class.

download, extract and run ant.

Play around with the parameters to see the affect. Generally it seems to thrash more with increased concurrent threads and interestingly in jdk1.6 sometimes reverts to the default timeout instead of the instance value.

Based roughly on the following logic:
When Circuit is Closed:
on call = pass through
call succeeds = reset count
call fails = count failure
threshold reached = trip breaker. Open State

when Circuit is Half-Open
on call = pass through
call succeeds = reset go. Close State
call fails = trip breaker. Open State

when Circuit is Open
on call = return/fail
on timeout = attempt reset. Half-Open State

Tweet This Post

System Management Patterns Overview

Collect Correlated System Metrics

Gather and store correlated system metrics.

The essence of this pattern is enable collection of multiple correlated data points over time to provide a picture of the system components that can be analysed at all interesting layers. Enabling better informed business, architectural and design decisions about the system or it's components.

The necessity of this pattern grows as we handle the web's bursty traffic nature and rely on more distributed services. I cannot over state the significant value this provides that will flow through the organisation and it's partners.

How it Works

Identify what components make up your system, what dependencies there are and start from the bottom of the pyramid.

There are many mature tools and protocols that already provide monitoring at the Operating System level.

Depending on what container or virtual machine you have they may support the defacto industry standard for monitoring.

Track major data points within your application. For a Web based retailer there are particular areas that would be important, such as Inventory Searches, Purchasing, third party dependencies, like Payment System Providers.

Using It

• Collect correlated system metrics in real-time. Log file analysis has a longer feedback loop, can introduce correlation issues, easily lost or overwritten and doesn't scale well as more system components are added.
• Use it often, resolve problems with collection as a high priority, the value it provides decrease as reliability of results or collection.
• Prefer the collection of raw data points over aggregated data where possible as you need to make arbitrary choice of aggregation which may prevent the use of the metric with other metrics. If you were measuring response time for a service serving many concurrent clients, you could track, hitcount and total response time or you may track hitcount and avg. response time over what: period of time, or last x hits?>
• Think about coverage, do you have too much? Do you have too little? Don't get too much information, avoid paralysis by analysis.
• Think about your system, it is alive, it constantly changes, whether that is code, hardware, os or business usage, everyday is unique!
• Don't blindly trust the statistics you have, occasionally, independently verify important statistics, does not have to be exact, could use back of the envelope calculations.
• Serious usage may warrant a separate network.

Example

The technology to support the implementation of this has been around for years
and is mature in the infrastructure layer of a system. There are several network management applications that build upon SNMP and Round Robin Database Tool to collect real time data and specify a suitable granularity of aggregation to limit archive size, yet still allow queries to be performed.

SNMP monitoring has recently been added to the Java JVM itself. At the application level there are several snmp implementations that could be used to enable a poller, or statistics collector to query your app for certain metrics.

So it is possible to create a system view that spans the Operating System (cpu, disk, network adaptors), Virtual Machine, Container and your application.

Reap the Benefits

Knowledge of your approximate workload is the first step.

The benefits flow from the day to day operational aspects to the CIO assuming sufficient coverage has been implemented. Operations/Production Support have an increased ability to diagnose issues, they can identify cause and affect immediately after a change instead of guessing, to the strategic end of town. Better diagnosis of issues, may only be to the extent that significant time is not wasted looking in an area that is not causing the problem. In many cases it will highlight what component is or is not causing a problem and further analysis of the Correlated Component Snapshot needs to be investigated.

Developers can use those numbers to approximate workload and test large scale changes to the system, like changing operating systems, changing databases, doubling inventory etc.

• CIO, can get more confident answers to:
  What is the load sensitivity to the system, if we double inventory? What happens if we introduce a promotion that is likely to increase traffic 10 times for the length of the promotion? How much hardware do we need to purchase to support these loads?
• Technical Operations Manager, are my servers up? What happened at 13:00, to spike cpu usage by 20%? Is load within reasonable? limits? Are there errors on any devices? We changed disk arrays and performance has decreased?
• Production Support, should have a dashboard of important points usually spanning all layers. They can ask questions like: Hey we requested a that package xyz be upgraded and response time of service abc has doubled?
• Development Team, the real workload is alot different to what everyone anticipated. Do we need to focus on a different area? Do we need to add or tune caches?

Correlated Component Snapshot

Collect correlated detailed information of a System Component.

How it Works

Periodically collect detailed correlated information about your System Component . Let you travel back in time to see what happened to a component, maybe even to see the cause of the resulting affect. It is useful for diagnosing system problems in both production as well as determining and removing bottlenecks when Scaling Approximate Workload.

Using It

• Gather and store detailed correlated information of a System Component.
  Collected periodically based on a balance of component usage, gathering cost. 20 minute i is a good rule of thumb
• A listing of top running processes at that point in time.
• A database server, may have information such as locking transactions, long running transactions, database specific internal statistic reports.
• Typically recent data would be stored. Older data to be archived or removed.

Scaling Approximate Workload

Build tests that can approximate the real workload or your system.

How it Works

Having the ability to scale an approximate workload of your system that is monitored via Collect Correlated SystemMetrics provides the ability to really see what is happening to the system when changes occur. It provides the final metrics by which changes to components or workload can analysed and simulated.

This enables developers to simulate and answer "What If?" type of questions.

Using It

• Build read only tests and read-write tests of vital functions. Read-only tests will provide a further level of approximation without the extent of data management imposed by the read-write test.
• The tests must be written in a loosely coupled way, to enable simple scaling up of workload. They are intended to be used by a tool that allows simple manipulation of scaling parameters, such as number of clients, number of hits per hour etc.
• Approximate data usage as well as functionality usage. While randomness of data is important, significant data distribution should reflected in you most important tests. Don't randomly choose products if a product category represents significant proportions of browsing or searching. If 50% of products browsed are from 1 product category then weight it accordingly.

Ping

Is your service platform alive and what is the upper bound performance limit [Bulka] I could potentially achieve from a service on your platform at this time. This gives meaning to the efficiency of system components under real or approximate workloads when monitored via Collect Correlated System Metrics

How it Works

A simple service of your service platform that does nothing, but returns immediately with a suitable response. Requires that a Ping service on the service platform be implemented.

In the case of a Servlet this may be an empty html form with a 200 response code. In the case of a webservice it responds in a similar way, perhaps with application specific response codes as well. In the case of a virtual machine you may be pinging a synchronized resource, to check it's liveness [Lea].

Using It

• This can be used for internal components and more beneficially for external third party components.
• Very effective in helping to determine where limiting factors lay; virutal machines, operating systems, design or architecture of a system
• Dynamically adjust your system to reduce workload on third parties. As webservices adoption grows businesses will become even more reliant on responsiveness of third party partners. In an ideal world any third party systems will be able to handle any load your system can throw at it. But in site that is growing exponentially, partners may not have the inclination, contractual obligation or capabilities to scale as fast as your organisation. It is most likely the case that some messages are more important than others and that changing the workload you place on their system, or altering practices to accomodate cyclic degradation in their system could be very beneficial.

References

[Bulka] Java Performance and Scalability, Volume 1

[Fowler] Patterns of Enterprise Architecture

[Lea]
Concurrent Programming in Java(TM): Design Principles and Pattern (2nd Edition)

Tweet This Post

The current project I'm working on is slowly replacing an existing system and one of the worst smells we've come across is the mixing of Reporting and Data Entry functions in an application.
Firstly the Searching/Reporting is time consuming (5-60secs+) full table scans on their order tables including filtering on obscure low cardinality columns which were often derived or dependant on child table row existence...

The table in question is the primary order entry table for the business, so it has alot of contention already.

To make matters worse they are currently using SQL Server which has a proven incredibly useless locking mechanism. We have tests that prove that readers block writers when competing for the same resouces.

The old system reduced the affect of some of these limitations by using READ_UNCOMMITTED Isolation Level alot.

It appears as though they tried not to think about any sort of locking, pessimistic or optimistic.

It's been a slog to get business to give up what they already have been given, but Greg is very persuasive.

Some prior work had already come across the Reporting issues so there was already a Reporting Server that has data upto a week out of date.

All reporting aspects of the story could be moved over to the reporting service. We agreed on a date range that our system would search over and for searches prior to that could be performed on the reporting server.

That was most of the problem solved, the other issues are why our project started in the first place.

Tweet This Post