Another good event, great speakers, lots of language talks, agile, architecture, scalability, databases ala megastore. highlights for me, in no particular order.

• Avi Bryant's new project, great eye candy had the crowd go "ooooooooohhh". When will an api be open to plug other data in???

  • Also nice talk on VM history, how we are working on VMs from algo's designed in the 80's. And the features or standard of VMs aren't available for all our favourite languages.

• Tiersen , good intro to sharding and google approach to scaling out

  • liked the probability graphs
  • read chubby, logserve paper
  • higher level of jonas discussion, but jonas did refer to tiersen counter example
  • When the web came along, our apps were read mostly.
  • These days, social sites, are moving away from this scheme, to one of shared data. Some of that shared data is written to a lot. but it is updated, the majority of it is always inserts only.
  • write fan outs.
  • combine the two operations in one, so write + read all counters at same time
  • counter example too simple, can bypass optimistic locking as counter should be monotonically increasing
  • combine websites with fragments of data/entities with different shards and shard policies.

• Jonas - great talk re Megastore

  • Treat everything as an insert, duplicates will occur make sure they are idempotent
  • Updates are a little harder, but what is an update really, when can you say it is actually done?
  • Confirmed for me that a current strategy i'm implementing is right and will work (well for two, maybe three nodes anyway).
  • To get to more will need a decent Paxos implementation, which takes smart ppl and time.
  • consistency, availability, entity groups and big table
  • web apps more read/write than before, lots of shared data, but mostly additions
  • consistency, reliability, storage, scalability and megastore scale, entity groups, transactionality, avoiding joins
  • consistency: user trust, none, eventual, entity group, global
  • consistency: "a harmonious uniformity of agreement among things or parts"
  • Sharding by entity groups.
  • storage vs. cost of loss
  • paxos vs. 2pc
  • papers jonas recommended: pat helland paper; jinquan dai, intel, james hamilton, MS
  • I have a lot more to say about this topic, as I cut my teeth on Oracle Performance Tuning and it's architecture, always providing a READ CONSISTENT transaction at a minimum, I use to think was a great idea, until I wanted to scale it further.
  • Disk Reads are about 10,000 slower than memory access, but not if you have to manage a lot of versions of different blocks in memroy. The overheads reduce the read in memory to ONLY 10 to 100 times faster than a disk read. That just isn't enough. See Milsap papers on Oracle scaling there are a number of them and James Morle's book on Scaling Oracle 8i which is a great book, the older print version can still be picked up as well
  • The approach of one single db instance creates new problems, now we need transaction logs and a hot standby and a DR data centre...

• Nygard (Release It! )- i missed the first one due to a conflict <img src='http://brettdargan.com/blog/wp-includes/images/smilies/icon_sad.gif' alt=':(' class='wp-smiley' /> .

  • Great stuff from what i hear, I've been pushing for some time implementation of his patterns.
  • motivated a number of devs at my company. Hopefully I'll see some badly behaving webservices and stability pattern implementations.
  • Here is a simple Java Circuit Breaker Pattern Implementation from a ways back.

• mike cannon-brookes - had to fill a tough set of constraints, but interesting story about atlassian.

• Clemens

  • ms unified component thingy. simliar to osgi. Nice talk about issues with design, component composition and how all of IT boils down to composition of things at varying levels.
  • Discussion about component composition and how state is always a problem, yes.
  • Advocate of service use, not reuse as it should be used "as is"
  • I prefer service use over code/component, especially with a services developed with RESTful intentions

• Dan North, telling project war stories based on experience and * observations of good architects * soa gone bad, wsdl

  • Listen, Listen, Listen
  • technical problems aren't the biggest issue, silo communication
  • replacement of tools sqlserver to oracle, not solving the real problem
  • the nameless quality
  • vision, inspiration, enabler
  • project shaman
  • empathise
  • self belief, a sense of conviction and humility

• Joshua Bloch - great stuff

  • checkout MapMaker

• Eastman - apache mahut. Dirichlet clustering, hmm, that algorithm wasn't in Collective Intelligence

• Douglas Crockford

  • javascript inspired by self, scheme, perl and java
  • prefer === over == doesn't do type coercion
  • he no longer uses ++ and -- anymore, implicated in buffer overflow exploits
  • lambda, dyn objs, loose typing and object literals.
  • refreshing to discuss languages and language features again
  • the memoizer example was good, check the slides for recursion usage
  • use functions to make objects
  • functional inheritance
  • be rigorous and use jslint

Anyway I liked the Circuit Breaker pattern, as I've seen a number of apps with highly coupled systems that barf once latency increases in the back ends then you get "Cascading Failures" and the webapp's request handler thread pool is exhausted and they all wait their mandatory 30 seconds for a response. Eventually the app either dies, or gets taken out of a load balance farm causing a "Chain Reaction".

Here is an Experimental Circuit Breaker Implementation (circuit_breaker.tgz) , NOT built or tested FOR PRODUCTION use in anyway. There is explicity no thread safety no the actual CircuitBreakerSimple class.

download, extract and run ant.

Play around with the parameters to see the affect. Generally it seems to thrash more with increased concurrent threads and interestingly in jdk1.6 sometimes reverts to the default timeout instead of the instance value.

Based roughly on the following logic:
When Circuit is Closed:
on call = pass through
call succeeds = reset count
call fails = count failure
threshold reached = trip breaker. Open State

when Circuit is Half-Open
on call = pass through
call succeeds = reset go. Close State
call fails = trip breaker. Open State

when Circuit is Open
on call = return/fail
on timeout = attempt reset. Half-Open State

Save some cycles, by using strace -p <pid>.

Leave these till after you have done your stracing:

• Checking what releases have gone out recently
• Kernel upgrades/dmesg errors
• if you have filled a a filesystem, ok df -h is pretty quick as well
• kill -3 <javapid>

SNMP is a natural fit for publishing application statistics. The thing I don't like about existing snmp agent implementations is that they are all pretty language dependent and they are built on the old style, define a mib, crank a wheel to generate a template, then you finally get to add code for you metrics.

So firstly work has been done to have an agent publish statistics based on results from an xml stream. This has been built on top of pysnmp.

This allows us to have an agent on each device/box that can publish stats, regardless of language as long as you instrument the device in question (and you have a python 2.4 environment).

Once your stats are published, you will need a manager to gather them.
While any SNMP manager can be used, often like with jffnms, you will need to write your own poller anyway. We've created an easily configurable pollers so a manager, can poll and update rrdtool database.

While some of the leg work is alleviated, you will still need to do the following: instrument your app, define your rrd datasources and optionally design your mib.

I'm currently looking at opensourcing this very shortly.

System Management Patterns Overview

Collect Correlated System Metrics

Gather and store correlated system metrics.

The essence of this pattern is enable collection of multiple correlated data points over time to provide a picture of the system components that can be analysed at all interesting layers. Enabling better informed business, architectural and design decisions about the system or it's components.

The necessity of this pattern grows as we handle the web's bursty traffic nature and rely on more distributed services. I cannot over state the significant value this provides that will flow through the organisation and it's partners.

How it Works

Identify what components make up your system, what dependencies there are and start from the bottom of the pyramid.

There are many mature tools and protocols that already provide monitoring at the Operating System level.

Depending on what container or virtual machine you have they may support the defacto industry standard for monitoring.

Track major data points within your application. For a Web based retailer there are particular areas that would be important, such as Inventory Searches, Purchasing, third party dependencies, like Payment System Providers.

Using It

• Collect correlated system metrics in real-time. Log file analysis has a longer feedback loop, can introduce correlation issues, easily lost or overwritten and doesn't scale well as more system components are added.
• Use it often, resolve problems with collection as a high priority, the value it provides decrease as reliability of results or collection.
• Prefer the collection of raw data points over aggregated data where possible as you need to make arbitrary choice of aggregation which may prevent the use of the metric with other metrics. If you were measuring response time for a service serving many concurrent clients, you could track, hitcount and total response time or you may track hitcount and avg. response time over what: period of time, or last x hits?>
• Think about coverage, do you have too much? Do you have too little? Don't get too much information, avoid paralysis by analysis.
• Think about your system, it is alive, it constantly changes, whether that is code, hardware, os or business usage, everyday is unique!
• Don't blindly trust the statistics you have, occasionally, independently verify important statistics, does not have to be exact, could use back of the envelope calculations.
• Serious usage may warrant a separate network.

Example

The technology to support the implementation of this has been around for years
and is mature in the infrastructure layer of a system. There are several network management applications that build upon SNMP and Round Robin Database Tool to collect real time data and specify a suitable granularity of aggregation to limit archive size, yet still allow queries to be performed.

SNMP monitoring has recently been added to the Java JVM itself. At the application level there are several snmp implementations that could be used to enable a poller, or statistics collector to query your app for certain metrics.

So it is possible to create a system view that spans the Operating System (cpu, disk, network adaptors), Virtual Machine, Container and your application.

Reap the Benefits

Knowledge of your approximate workload is the first step.

The benefits flow from the day to day operational aspects to the CIO assuming sufficient coverage has been implemented. Operations/Production Support have an increased ability to diagnose issues, they can identify cause and affect immediately after a change instead of guessing, to the strategic end of town. Better diagnosis of issues, may only be to the extent that significant time is not wasted looking in an area that is not causing the problem. In many cases it will highlight what component is or is not causing a problem and further analysis of the Correlated Component Snapshot needs to be investigated.

Developers can use those numbers to approximate workload and test large scale changes to the system, like changing operating systems, changing databases, doubling inventory etc.

• CIO, can get more confident answers to:
  What is the load sensitivity to the system, if we double inventory? What happens if we introduce a promotion that is likely to increase traffic 10 times for the length of the promotion? How much hardware do we need to purchase to support these loads?
• Technical Operations Manager, are my servers up? What happened at 13:00, to spike cpu usage by 20%? Is load within reasonable? limits? Are there errors on any devices? We changed disk arrays and performance has decreased?
• Production Support, should have a dashboard of important points usually spanning all layers. They can ask questions like: Hey we requested a that package xyz be upgraded and response time of service abc has doubled?
• Development Team, the real workload is alot different to what everyone anticipated. Do we need to focus on a different area? Do we need to add or tune caches?

Correlated Component Snapshot

Collect correlated detailed information of a System Component.

How it Works

Periodically collect detailed correlated information about your System Component . Let you travel back in time to see what happened to a component, maybe even to see the cause of the resulting affect. It is useful for diagnosing system problems in both production as well as determining and removing bottlenecks when Scaling Approximate Workload.

Using It

• Gather and store detailed correlated information of a System Component.
  Collected periodically based on a balance of component usage, gathering cost. 20 minute i is a good rule of thumb
• A listing of top running processes at that point in time.
• A database server, may have information such as locking transactions, long running transactions, database specific internal statistic reports.
• Typically recent data would be stored. Older data to be archived or removed.

Scaling Approximate Workload

Build tests that can approximate the real workload or your system.

How it Works

Having the ability to scale an approximate workload of your system that is monitored via Collect Correlated SystemMetrics provides the ability to really see what is happening to the system when changes occur. It provides the final metrics by which changes to components or workload can analysed and simulated.

This enables developers to simulate and answer "What If?" type of questions.

Using It

• Build read only tests and read-write tests of vital functions. Read-only tests will provide a further level of approximation without the extent of data management imposed by the read-write test.
• The tests must be written in a loosely coupled way, to enable simple scaling up of workload. They are intended to be used by a tool that allows simple manipulation of scaling parameters, such as number of clients, number of hits per hour etc.
• Approximate data usage as well as functionality usage. While randomness of data is important, significant data distribution should reflected in you most important tests. Don't randomly choose products if a product category represents significant proportions of browsing or searching. If 50% of products browsed are from 1 product category then weight it accordingly.

Ping

Is your service platform alive and what is the upper bound performance limit [Bulka] I could potentially achieve from a service on your platform at this time. This gives meaning to the efficiency of system components under real or approximate workloads when monitored via Collect Correlated System Metrics

How it Works

A simple service of your service platform that does nothing, but returns immediately with a suitable response. Requires that a Ping service on the service platform be implemented.

In the case of a Servlet this may be an empty html form with a 200 response code. In the case of a webservice it responds in a similar way, perhaps with application specific response codes as well. In the case of a virtual machine you may be pinging a synchronized resource, to check it's liveness [Lea].

Using It

• This can be used for internal components and more beneficially for external third party components.
• Very effective in helping to determine where limiting factors lay; virutal machines, operating systems, design or architecture of a system
• Dynamically adjust your system to reduce workload on third parties. As webservices adoption grows businesses will become even more reliant on responsiveness of third party partners. In an ideal world any third party systems will be able to handle any load your system can throw at it. But in site that is growing exponentially, partners may not have the inclination, contractual obligation or capabilities to scale as fast as your organisation. It is most likely the case that some messages are more important than others and that changing the workload you place on their system, or altering practices to accomodate cyclic degradation in their system could be very beneficial.

References

[Bulka] Java Performance and Scalability, Volume 1

[Fowler] Patterns of Enterprise Architecture

[Lea]
Concurrent Programming in Java(TM): Design Principles and Pattern (2nd Edition)

I never really got into SNMP before and since Tiger directly supports it, the time had come.

The aim was to add our JVM to an existing monitoring system, jffnms.

Being a newbie to SNMP I had a few setbacks than anticipated.

While the jconsole is cool and useful, I wanted application/JVM monitoring to fit in with our existing monitoring tools and also for collected data to be nicely correlated with os and network stats.

Sun's SNMP guide to enabling a port for monitoring is straight forward, but I found learning about snmp from google painful, and the best bits were hard to find.

If you've got/done the following then it will be easy to get started:

1. net-snmp installed and read some doco on SNMP and MIBS, the net-snmp doco is better than most but has alot of fluff.
2. tiger
3. have added followed the snmp monitoring setup so you have a listenter and configure acl.
4. and some kind of server to run

If any of the acl settings are wrong or file permissions aren't set correctly, the jvm won't start.

Once it is going you could double check, using netstat.

Now to use the snmp tools you will need to load in Sun's JVM-MANAGEMENT-MIB

It's good practice to use snmptranslate first to see if there are any problems with the MIB definition, in particular any missing dependencies.

snmptranslate -M .:/usr/share/snmp/mibs -m JVM-MANAGEMENT-MIB -IR -Tp jvmMgtMIB

Assuming you are running this from the current directory where the new MIB is that command should work. -M is giving a search path for mibs and -m is telling is specifying which mibs we want to load. The final parameter, jvmMgtMIB is the module identity, you can see that inside the mib file.

You should see a tree view of the mib starting like this:
+--jvmMgtMIB(1)
|
+--jvmMgtMIBObjects(1)
| |
| +--jvmClassLoading(1)
| | |
| | +-- -R-- Gauge jvmClassesLoadedCount(1)
| | +-- -R-- Counter64 jvmClassesTotalLoadedCount(2)
| | +-- -R-- Counter64 jvmClassesUnloadedCount(3)
| | +-- -RW- EnumVal jvmClassesVerboseLevel(4)
...

And finally we can peek at some jvm stats, just plugin your port:

snmpwalk -v 2c -M .:/usr/share/snmp/mibs -m JVM-MANAGEMENT-MIB -c public localhost:port jvmMgtMIB

So in this snippet we are using snmp version 2c and still loading the sun mib.

You should see some results like this:
...
JVM-MANAGEMENT-MIB::jvmMemoryHeapInitSize.0 = Counter64: 838860800 bytes
JVM-MANAGEMENT-MIB::jvmMemoryHeapUsed.0 = Counter64: 39430568 bytes
JVM-MANAGEMENT-MIB::jvmMemoryHeapCommitted.0 = Counter64: 832438272 bytes
JVM-MANAGEMENT-MIB::jvmMemoryHeapMaxSize.0 = Counter64: 1664876544 bytes
JVM-MANAGEMENT-MIB::jvmMemoryNonHeapInitSize.0 = Counter64: 8552448 bytes
JVM-MANAGEMENT-MIB::jvmMemoryNonHeapUsed.0 = Counter64: 9260472 bytes
JVM-MANAGEMENT-MIB::jvmMemoryNonHeapCommitted.0 = Counter64: 9863168 bytes
JVM-MANAGEMENT-MIB::jvmMemoryNonHeapMaxSize.0 = Counter64: 100663296 bytes
...

To make this more useful you should install the mib properly, put it in with the others and update your ~/.snmp/snmp.conf file to have "mibs +JVM-MANAGEMENT-MIB".

One of the things I hated about getting started with SNMP were the misleading errors.
Such as

• snmpwalk: Timeout - can be caused by a bunch of things, like not specifying the correct version of snmp that you are trying to query!
• SNMPv2-SMI::mib-2 = No Such Object available on this agent at this OID - can be caused when you don't specify the OID, eg. jvmMgtMIB

Well writing this out takes too long, I'll have to figure out the jffnms integration later.