100 Words

Posted on Thursday 28 October, 2010 in forward

Paul and I had to write a 100 words to describe Forward when we sponsored the first clojure conj a.ka. the conj. This is what we came up with:

Forward is a collection of internet companies based in Camden, London. Our success is driven by talented people who are given the freedom to experiment with technology in a low ceremony environment. Forward is reliant upon small teams of developers who actively engage in the business and go beyond a traditional role. Development is at the core of how we solve problems, from testing new businesses in a day to discovering how the weather affects web traffic. This has made us one of the top ten fastest growing privately owned technology companies in the UK.

Sound interesting? Get more information at http://forwardtechnology.co.uk/ or get in touch

3 Cool Ruby Tricks

Posted on Wednesday 27 October, 2010 in ruby

Destructing arrays

Splat a range into an array

Mass assign variables from a hash with values_at

Building Application Level Abstractions

Posted on Thursday 14 October, 2010 in programming and domain-driven-design

The abstractions we build to solve a problem are just tools and, as such, should be created for utility not to match reality. A problem can be solved in any number of ways and when delivering software it is no different. You should tailor the abstractions you create to solve your specific problems.

When an application address a number of problems instinct often suggests a single abstraction that either models the real world is as “flexible and generic” enough to solve any problem in the domain. However in trying to unify the solutions in either of these ways leads introduction of complexity. Generalising a solution makes it less tailored to solving an specific problem. The result is often a confused code base, an overloaded language used to describe problems and (perhaps more importantly) users having to deal with the limitations of them system.

Bounded Contexts

In his book, Domain Driven Design, Eric Evans refers “bounded contexts” as being logical groupings around parts of an application focused on solving a distinct problem. In these bounded contexts a common language and understanding of the domain often exists. Acknowledging the existence of these contexts and introducing distinct boundaries allows focus on delivering solutions that are tailored to individual problems. The resulting clarity from appropriate abstractions and communication in a common language allows for more effective solutions. It often manifests itself though much simpler code, faster delievery and a solution for the user that not constrained or impeded by the system limitations.

Implications of a Single Abstraction

Recent experience of how effective this technique was shown in delivering an application that provids price comparisons for various home utilities. There are where distinct parts of the system: the import and administration of data, comparison and pricing of data relevant to a user, and presentation of this data to a user.

Previously there was a single model and language for each of these areas and even in such a simple application it had a profound effect. For administrators it was a thoroughly frustrating experience to import the data as it took them hours to complete a simple tasks. For developers it resulted in a code base in which parts were untouchable and a system that was inflexible and difficult to make anything but the most superficial of changes in. For an end user data was often out of date or a system that wasn’t responsive.

Results of Modular System

Given the opportunity tackle this legacy system allowed the introduction of bounded contexts around the three areas and to evolve within them seperate, tailored, abstractions. This quickly achieved some impressive results.

Having a specific language around data administration and an abstraction that was relevant to the task (not required conform with over arching model) allowed for a number of profound changes. The users had a system that was simple to use and solved their problems. It reduced the time for common tasks down to minutes from hours and days. The abstractions of the other parts of the application did not leak into it resulting in a simpler code base in which changes and improvements could be be made quickly and by anyone on the team. A common language with which to discuss issues with the users allowed for much more effective communication around new features.

In other areas being seperate from the rest of the system allowed for experimentation. A different language was introduced in one context that was much more suited to the problem of comparison.

The front end code base was often under high load which previously this meant scaling the entire system. Using this modular approach meant the concerns of each system could be addressed in isolation. This lead to a more performant application at a fraction of the cost.

Implications of Modular System

In introducing different contexts, each with their own abstractions, the obvious implication is that some translation is required when they communicate. Its important to note that this mapping has always existed but was either applied mentally, or worse, in the leaked into UI. The ability to contain this mapping programatically at the boundaries of the applications means frees you from making compromises in other areas.

Splitting an application into distinct modules may increases the amount coordination required to deploy the application. However this is easily surmountable using tools like capistrano, upstart, nginx, AWS etc.

Conclusion

Solving any problem effectively means acknowledging that it can be solved in a number of different ways and finding the right approach. Delivering an application means indentifying each of the distinct problems that exist and finding or evolving abstractions for each. With the right abstractions in place one can easily solve problems within this domain to quickly deliver features and produce a more effective and performant application.

Enabling Experimentation

Posted on Monday 04 October, 2010 in programming, domain-driven-design, and experimentation

The slides for the presentation that Paul, Nick and I gave at the first Forward Awayday.

Working with JRuby

Posted on Thursday 08 July, 2010 in programming, jruby, RMV, and NailGun

It was difficult to get an effective work flow when we first chose to use jruby for part of our application as nothing was integrated and everything ran a bit slowly. With the help of RVM and Nailgun we managed evolve something that made development almost as easy as using MRI.

Automatically switch to JRuby

As you switching back and forth between projects using different ruby version it quickly becomes a pain having to manually specify the correct one. The project .rvmrc removes the need to do this as when “cd” into your project directory it automatically switches to the correct ruby version. The example below creates a .rvmrc file for a project that uses jruby with the spreadsheets gemset.

local$ echo “rvm jruby@spreadsheets” > .rvmrc

This works when using ruby from the shell but unfortunately TextMate will still be unable run any of your projects files or tests but there is easy to fix.

Configure TextMate’s TM_RUBY variable

RVM again helps by allowing you to create a wrapper script that you can configure TextMate to use instead of the default ruby. To create a script that uses jruby and the same gemset as above run the following command.

local$ rvm —wrapper jruby@spreadsheets textmate

A wrapper script is then created in ~/.rvm/bin/textmate_ruby. Now in TextMate you can create a TM_RUBY shell variable (Textate > Preferences > Shell Variables) and point it at the fully qualified file path. Now cmd+r will run any files or tests form from TextMate again.

This would have been enough it it were not for the JVM startup time. Each time you have to wait those few extra seconds before it your test run it seems longer and longer.

Speeding up Tests

The best solution I for getting the tests to run faster is to drop to the command line and use Nailgun to cut out the start up time. Fire it up with:

local$ ruby —ng-server &

Now you can run your tests with the —ng flag to cut out the start up time (well after the first run anyway). For example if you were using RSpec then simply run:

ruby —ng -S spec path/to/your_spec.rb

If you just want to run a single test then just specify a line number:

ruby —ng -S spec path/to/your_spec.rb -l 12

HTH

Moving to Jekyll

Posted on Friday 25 June, 2010 in programming and blogging

Recently I decided to move my blog from Wordpress and try to find a simpler way of maintaining a blog. Wordpress has always felt a bit overkill when my only real requirement was an easy way to generate consistently styled HTML pages.

James Mead had previously pointed me in the direction of a number static site generators and after some googling I was surprised to find how many options there actually were. After playing around most of them I quickly narrowed my options down to:

  • webby: super simple CMS
  • staticmatic: inbuilt HAML and SASS support
  • nanoc: lots of features but didn’t like the configuration
  • jekyll: blog aware static site generation

Jekyll was the seemed closest fit to my requirements and had the easiest setup and I have been playing with it for about a month or so. Some of the features of using a static site generator that I have really liked are:

  • The posts are stored in source control and not a database
  • I can use my favourite editors to edit any file.
  • Easy to modify, change or totally redesign your site on a whim.
  • The source is in ruby so I can can easily hack at it.
  • Forks providing HAML / SASS support.

The only feature that was missing from Wordpress was commenting but thats easily solved with disqus.

So if you are currently running your own blog on Wordpress and want something fast and light weight I would encourage to try it out.

15 Applications in 9 months

Posted on Sunday 20 June, 2010 in forward and process

Image from striatic’s flickr stream under the Creative Commons licence

Paul and I recently presented a short overview of how our team of 6 developers was able to deliver 15 applications in the last 9 months. The aim was to highlight the processes our team have evolved during this time and challenge preconceptions about how software can and should be delivered. This Post includes the information from that overview supplemented with many of my own opinions.

Team

Our team at Forward is by far the most experienced I have ever worked with, infested with ex-Thoughtworkers and balanced with members with complementary experience from agencies such as New Bamboo. The most important quality that all the developers share is that everyone really understands the web and has a real passion for technology. As a team we have adopted a very flat hierarchy and the company has shown its trust in us by giving us the autonomy to make our own technical decisions. Perhaps most importantly though we have Carl Gaywood, the definition of a motivated person to build an agile team around.

Projects

The projects we have worked on vary from smaller sites such as ukofficeadvisers.co.uk and officeinthecity.com (the latter being written, deployed, and trafficked in less than one day) to applications with a huge feature set like cardwall.co.uk. This application was inspired by agile card-walls and was created to help our agency teams manage projects and collaborate with our clients. Cardwall also provides an interface to many of the other applications we have written. These vary greatly in their requirements, from technical challenges such as spell checking AdWord campaigns across many languages, to large scale problems like downloading and processing tens of gigabytes of data per day. The latter requiring the co-ordinating analytical and statistical checks which run thirteen hundred times daily to provide intelligent notification.

As well as dealing with scale in terms of data and processing, we also needed to provide a service to handle and log the 9,000,000 daily clicks we receive. With an estimated cost of down time of £200 per minute, it has been architected with a geo-distributed high availability infrastructure. It also incorporates a phased deployment strategy that has meant it has yet to experience any downtime.

We are constantly spiking and building new applications. Recent tools we have created include a realtime visualisation of internet search trends and a custom interface to Hive allowing users to generate, run and even automate queries on this data. Alongside this, we have automated SEO checks, produced landing pages and even created an application to view Formula 1 pictures on a iPad.

What did we learn?

Being involved in such a large and varied number of projects in such a short amount of time has required us to be able move fast, maintain a high level of quality, and be able change direction quickly when things aren’t working out. We began with a more traditional agile approach but, in the spirit of constant improvement, have evaluated and evolved, and shed many processes. The following summarises a large part of these changes and highlights some values that helped us achieve this.

Experimentation

In developing these applications we used a wide variety of tools: key value stores; databases; map reduce; virtualised infrastructure; dedicated infrastructure; geo-distributed high availability infrastructure; Java; Ruby; HTML; CSS; Clojure; jRuby; and SQL. We don’t allow ourselves to get locked in to a technology stack, or vendor. Experimentation with tools,frameworks and languages allows for the possibilities of new ways to work, think and solve problems.

Small teams, flat structures

With each application we often split the main team into pairs who manage and run each project. The more people and hierarchy on a project, the more processes are required to keep the team effective. For example, project inceptions are now a 20-minute discussion not a 2-week session without any code. Smaller teams also don’t need processes to aid communication: stand-ups, IKO’s or retrospectives just aren’t necessary when you work constantly with the only other person on the project. Our small, empowered teams, focused on delivering business value, could easily outpace any other teams I have worked in.

Many Hats

An interesting constraint introduced by small teams is that you can no longer just code. You need to be able to talk and interact with other areas of the business to gather and prioritise requirements. Good ‘nix and system administration skills are required to setup and configure servers so any application can be deployed and monitored. Strong database skills are also often required, understanding indexes, query optimisation, and often, in our case, many of MYSQL’s nuances. This self-reliance speeds delivery up as any problems are solvable within the team itself. It also has an added benefit in that it expands the number of ways the team is able to approach and solve a problem.

Test Driven Development?

Maybe the most controversial thing we have done is to stop test-driving all aspects of the application. TDD is a great tool, can help you decompose problems properly, provides a safety when refactoring, and is a great training tool but there are trade-offs. It slows you down and a large test suite can easily stop you from being able to change direction quickly. The lessons learned from TDD can also be applied without dogmatically following the TDD mantra. It is important to choose a level of testing appropriate to the application you are writing and the current phase of its life cycle. In our environment we have found testing the stress points, or having a few high level acceptance tests early on are all we need to allow us to move fast, maintain quality and change direction quickly.

The Rewrite

New technologies allow you to work in different ways. Throwing away code and starting again previously seemed cavalier, perhaps, even dangerous. As you work with a problem your understanding continually develops but from the implementation of the first feature you have started to limit your options in how you can solve it. Ruby has allowed us to use the knowledge of the problem and very quickly reshape an application to conform with our current understanding and this often involves throwing away much of the code.

Ownership

We also have a culture of real ownership of the applications created. Problems are tackled at any time, wherever we are Snowboarding in Canada or at home Christmas day. This has all been done without an SLA or any of us having to be on call. As a team we are dedicated to delivering quality and knowing that you will be actively supporting an application forces you to focus on providing notifications and details of errors when things go wrong. This has also extended to creating tools to monitor the health and status of the application and administration tools to visualise processes so we can easily see what is happening at any time.

Continuous Deployment

With a steady stream of requirements, a small team, and little process we are able to complete stories and deploy them as soon as they are finished. There is no need to wait for the end of an iteration as you can start getting feedback instantly.

These are a few of the important areas that I believe have contributed to our team becoming so highly performing and are also the key areas by which we differ from other teams I have worked with. Some are seemingly at odds with conventional wisdom but they all embrace the principles of the agile manifesto. I would stress again however, context is important and we have stripped away process that can be useful in many situations. You should always weigh up the potential benefits against limitations and drawback of any process or tool as part of a continually improving agile team.

About

A blog by Michael Jones, a developer currently working at forward, whose interests include: web technologies; ruby; functional programming; design and making the perfect cup of filter coffee.

Subscribe by RSS

Recently…