Another good event, great speakers, lots of language talks, agile, architecture, scalability, databases ala megastore. highlights for me, in no particular order.

• Avi Bryant's new project, great eye candy had the crowd go "ooooooooohhh". When will an api be open to plug other data in???

  • Also nice talk on VM history, how we are working on VMs from algo's designed in the 80's. And the features or standard of VMs aren't available for all our favourite languages.

• Tiersen , good intro to sharding and google approach to scaling out

  • liked the probability graphs
  • read chubby, logserve paper
  • higher level of jonas discussion, but jonas did refer to tiersen counter example
  • When the web came along, our apps were read mostly.
  • These days, social sites, are moving away from this scheme, to one of shared data. Some of that shared data is written to a lot. but it is updated, the majority of it is always inserts only.
  • write fan outs.
  • combine the two operations in one, so write + read all counters at same time
  • counter example too simple, can bypass optimistic locking as counter should be monotonically increasing
  • combine websites with fragments of data/entities with different shards and shard policies.

• Jonas - great talk re Megastore

  • Treat everything as an insert, duplicates will occur make sure they are idempotent
  • Updates are a little harder, but what is an update really, when can you say it is actually done?
  • Confirmed for me that a current strategy i'm implementing is right and will work (well for two, maybe three nodes anyway).
  • To get to more will need a decent Paxos implementation, which takes smart ppl and time.
  • consistency, availability, entity groups and big table
  • web apps more read/write than before, lots of shared data, but mostly additions
  • consistency, reliability, storage, scalability and megastore scale, entity groups, transactionality, avoiding joins
  • consistency: user trust, none, eventual, entity group, global
  • consistency: "a harmonious uniformity of agreement among things or parts"
  • Sharding by entity groups.
  • storage vs. cost of loss
  • paxos vs. 2pc
  • papers jonas recommended: pat helland paper; jinquan dai, intel, james hamilton, MS
  • I have a lot more to say about this topic, as I cut my teeth on Oracle Performance Tuning and it's architecture, always providing a READ CONSISTENT transaction at a minimum, I use to think was a great idea, until I wanted to scale it further.
  • Disk Reads are about 10,000 slower than memory access, but not if you have to manage a lot of versions of different blocks in memroy. The overheads reduce the read in memory to ONLY 10 to 100 times faster than a disk read. That just isn't enough. See Milsap papers on Oracle scaling there are a number of them and James Morle's book on Scaling Oracle 8i which is a great book, the older print version can still be picked up as well
  • The approach of one single db instance creates new problems, now we need transaction logs and a hot standby and a DR data centre...

• Nygard (Release It! )- i missed the first one due to a conflict <img src='http://brettdargan.com/blog/wp-includes/images/smilies/icon_sad.gif' alt=':(' class='wp-smiley' /> .

  • Great stuff from what i hear, I've been pushing for some time implementation of his patterns.
  • motivated a number of devs at my company. Hopefully I'll see some badly behaving webservices and stability pattern implementations.
  • Here is a simple Java Circuit Breaker Pattern Implementation from a ways back.

• mike cannon-brookes - had to fill a tough set of constraints, but interesting story about atlassian.

• Clemens

  • ms unified component thingy. simliar to osgi. Nice talk about issues with design, component composition and how all of IT boils down to composition of things at varying levels.
  • Discussion about component composition and how state is always a problem, yes.
  • Advocate of service use, not reuse as it should be used "as is"
  • I prefer service use over code/component, especially with a services developed with RESTful intentions

• Dan North, telling project war stories based on experience and * observations of good architects * soa gone bad, wsdl

  • Listen, Listen, Listen
  • technical problems aren't the biggest issue, silo communication
  • replacement of tools sqlserver to oracle, not solving the real problem
  • the nameless quality
  • vision, inspiration, enabler
  • project shaman
  • empathise
  • self belief, a sense of conviction and humility

• Joshua Bloch - great stuff

  • checkout MapMaker

• Eastman - apache mahut. Dirichlet clustering, hmm, that algorithm wasn't in Collective Intelligence

• Douglas Crockford

  • javascript inspired by self, scheme, perl and java
  • prefer === over == doesn't do type coercion
  • he no longer uses ++ and -- anymore, implicated in buffer overflow exploits
  • lambda, dyn objs, loose typing and object literals.
  • refreshing to discuss languages and language features again
  • the memoizer example was good, check the slides for recursion usage
  • use functions to make objects
  • functional inheritance
  • be rigorous and use jslint

Tweet This Post

I finished reading Neal Fords book "The Productive Programmer" [Ford2008] It wasn't quite what I expected, but none the less it was well worth the purchase.

While there are a few topics I agree with but not to the same extent as him, the topic I strongly disagree with is on the generation of large amounts of documentation.

Documentation in this form merely serves as another one of those checklist items on a project plan.

"Conveying understanding" should be the underlying purpose for which usually wiki pages or documentation is produced in order to communicate with others through time and space. Documentation is not just about information.

Consider your own experience when it comes to picking up documentation produced by others no longer working on the systems.

Most documentation is brain dumped with little thought given to the actual writing or conveying of information. Anyone can describe the complicated in a complex manner, it takes more skill to take a complicated details and make them simple to understand.

Large volumes of javadoc, don't help me, it is another layer of indirection and as Robert C. Martin says regarding code, "all comments are lies". Deep down everyone knows this, even though many people are still in denial.

Rarely did I get documentation that was:

• reasonably accurate in terms of what it was meant to do
• mostly it differed significantly in how it was actually implemented
• usually had typos in crucial lines that didn't aid understanding at all
• large gotchas aka time-bombs are not mentioned
• building and configuration where slightly covered
• the best you usually wish for is some clues for which you can pull out your best Dr. Holmes hat and get to work with

The choice of conveying understanding via documentation is one of the least effective mediums for communication. Refresh your memory of Alistair Cockburn's [CockburnAmbler]; analysis of various communication mediums and it's effectiveness.

Tweet This Post

If there was would you prefix your statements/queries instead of postfix?

Tweet This Post

To avoid wasted effort, I'll need to have wordpress blogs/articles using this format.
To get this installed on site5, was a little bit of a pain.

mkdir ~/bin

download docutils.sf.net
mkdir -p ~/bin
untar in ~/bin

python setup.py install --prefix $HOME #no sudo required

.bashrc, PYTHONPATH=$PYTHONPATH:$HOME # so you can test on cmdline

try a test value, rst2html.py <filename> should work.

solve any problems their first.

Now onto wordpress, install the rest.php plugin at launchpad.net.

Edit settings at the top of the file, for $prefix which needs to be the same value as setup --prefix and $rst2html to rst2html.py

Now I was stuck on the problem with wordpress executing it with environment variables that don't seem to include the PYTHONPATH that I require.

so a cheap hack around that is to just dynamically update the python search path in rst2html.py

sys.path.append('/your/python2.4/site-packages')

Now I have proper results being displayed, except the mode line, which marks the entry for processing by the rest.php filter, is also being processed.
-*- mode: rst -*-

This needs to be filtered out, either during the rst2html process or after,
so I went with just embedding it within a reST comment and all is well.

.. -*- mode: rst -*-

Tweet This Post

After spending Friday assisting with an internal Architecture conference at The Swiss-Grand on Bondi Beach.

Things where going well, until the end of the day that is. Here is a long history of events:

• 1730 - sms notification our 7pm Flight Cancelled
• 1815 - next flight is 9pm
• 1945 - Electrical Storm Hits, buckets down
• Hail taxi, didn't take too long, except for treading water to get bags in the back
• 2030 - board our doomed flight
• 2115 - our plane is broken and we get kicked out
• 2215 - at the other end of the airport we get shuffled on to a shiny new plane, when does that takeoff curfew end???
• 2300 - crap the curfew is till 11pm, we are still on the runway, and my socks are really wet
• phew, there is some other runway we can use that goes over the bay
• boo, now it's windy
• boo, now planes need to land
• boo, now it's still windy
• 23:?? - everyone gives up, we're screwed
• 00:25 - we are back at the terminal and were stuck in the entry walkway, the doors are locked shut somehow and they can't open it up...
• 00:40 - ok, we're inside. hmm, what is going on now, will virgin be finding us a hotel, now, since we are out of towners and they are partly responsible - No apparently not, they recommend we all line up and stay in the airport unless you have some friends you don't mind waking up and as long as you don't mind paying for a cab. 300 people, 1 staff member to reschedule everyone, that's going to be quick
• hmm - we weren't the only plane, there are a few others around and about 3 qantas planes as well
• 00:50 - we grab a cab, the first 3 hotels near by are chock full, apparently ppl have been grabbing accommodation for hours, BOO
• We start ringing ahead for known hotels, hmm, still full
• yay, finally we have a bed
• 01:48 - yay sleep time
• 07:30 - I ring virgin and arrange for another flight, bit of a problem, as far as they know I boarded
• First flight I can get out is 1300, boo
• 10:45 - Surely Virgin will give us complementary Lounge visit for our troubles
• BOOO!!! - Virgin says No
• Thumbs down to virgin

Tweet This Post

Top Tips applicable to me

from Hot Text: Writing for the Web

• People don't read they scan, so chunk your data and emphasize
• Cut your content, repeatedly, cut your content, you can write it in less.
• Put your conclusion at the top
• Write in active voice
• Keep paragraphs short and single themed
• Put links at the start or end of sentences
• Don't make people think: don't use large words and use ambiguouswords carefully

Review

Overall I've found this book quite useful. I didn't like the Object-Oriented analogies or the outdated and incorrect SEO advice.

The information was not always new, but it provided different perspectives with recommendations that were valuable. As a web user I scan text constantly everyday, yet somehow I ignored this when writing.

Other Online Copywrite books

I can't remember why I ended up getting this book over any of the others on my shortlist:

• The Online Copywriter's Handbook : Everything You Need to Know to Write Electronic Copy That Sells
• Net Words: Creating High Impact Online Copy

Tweet This Post

Which downloads mypkgname from <sys|int|uat|prod> yum repository with all of it's dependencies.

That obviously wouldn't always be possible, but it is achievable with only a little pain .

Use RPM for managing components of your System

RPM is not ideally suited to deploying java applications and all of it's dependencies, but it can work.

I've used it to package up an installation of activemq, with my application config and start/stop scripts.
This was then extended to have ant build it so we could eventually have cruise create rpm artifacts for a project which could eventually get automatically sent to a yum repository.

RPM is not that great but when you have to wrestle with RedHat AS atleast it gives us

• Dependency Management. Requires that mysql >= 5.0 must be installed.
• Hooks for pre/post install/uninstall Scripts. %postin let me create a os user if required, set appropriate permissions, create a database, db user, add tables and insert data.
• Ability to relocate the package on installation

Creating yum repositories sounds easy, but I haven't gotten to this yet.

What to do

1. Create your ~/.rpmmacros file
2. Create a specification file, don't forget it will has access to some of it's own types of variables that are defined in .rpmmacros file.
3. package up your application of choice in a mypkgname-version.tar.gz. This is your main input into rpm, Source0: in your spec file must exactly match this file
4. rpm -bb mypkgname.spec
5. check your pkg info with: rpm -qpi mypkgname-version-noarch.rpm and your list of files with rpm -qpl mypkgname-version-noarch.rpm.
6. sudo rpm -i mypkgname-version-noarch.rpm

Tips

1. Understand %files .I found it easier to build my own file.list, by using %files -f file.list. Defaulting attributes with %defattr(-,name,group) and %config to specify your config files are good to know about.
2. You don't have to use the root owned default database, you can specify your own with the --dbpath option
3. How can you tell if a package is being installed or updated?
   $1 of the scriplets (%pre, %postin...) contain the number of pkg's affected, so during an update you affect 2 pkgs, as it is an uninstall, followed by an install operation.
4. Scriptlets share the same environment, but that environment is special. We had issues running pre-packaged scripts that depended on env variables setup in .bashrc becaused it exited if it wasn't an interactive shell. ie. [ -z $PS1 ]
5. Keep multiple copies of ~/.rpmmacros files and replace the one you are interested in
6. If you are packaging ruby, python or java applications, then specify it as non architecture specific
7. Make it Relocatable if possible
8. Someone had a problem with BuildArch: noarch on FC4 x86-64, that noarch was not recognised as a valid architecture. We needed to add a line to some file in /var/lib??. Moral of the story is to try not to use debian based distro instead.

If you know almost nothing about RPM

• RPM can be used to manage src (c) or binary rpms.
• It works by unextracting a tar of the source/binary then building and executing scriptlets at the right moments.
• configuration files, identified by %config, like /etc/mypkgname are renamed when you erase a package to mypkgname.conf.rpmsave.
• Relocatable pkgs require all your packaged files to have the same prefix.

Thanks to co-conspirators Ajit George, Sam Jordan, Peter Klein, Robert Watkins.

Tweet This Post

Then you can insert values without getting them converted to latin on the client.

Too bad ActiveRecord::Base doesn't already have a hook for these on start queries, since you need to issue some query to set character set properties for just about every database anyway.

Tweet This Post

Today I had to try and recover about *4 months* of images for a family member from a SD memory card. That's Xmas photo's, first day at school etc.

Initial symptoms were "Naming Error" reported on the Canon 610. Reported as a E51 error on some models.

I wasn't worried at first, expecting a partition table corruption given the numerous reports, but after the second tool failed to identify any files I was getting really worried.

I tried alot of tools a couple of freeware, but most were commercial trials, but only Zero Assumption Recovery was successful. Probably retrieved about 85% of images back, some were missing, some of them only the thumbnails were retrieved.

Tools that didn't cut the mustard:
"PC Inspector smart recovery", "cardrecovery","flash file recovery", "data doctor recovery" "stellar phoenix DMR".

I'm pretty disappointed at the number of reported issues of memory card errors in a brand I was pretty happy with .

Atleast at some people have found a solution that works for them, but if it strikes again, try out Zero Assumption Recovery, the Image Recovery process is freeware.

Tweet This Post

My latest project is complete, only a couple weeks late due to insufficient calendar time.

Completed just before winter, our happy customer checks out the new digs.

Tweet This Post