Review of Release It!

Release It! Design and Deploy Production-Ready Software by Michael T. Nygard, published by The Pragmatic Programmers.
ISBN: 978-0-9787-3921-8

Introduction

Release It! is a book I have had on my reading list for a few years. I started a new job at Tradera/eBay Sweden in June last year and Release It! felt more relevant to my work than ever before. It’s the first time in my career that I am working on a public site with a lot of traffic (well, a lot for Sweden anyway). Downtime can mean a lot of lost revenue for my company. As I’m on call every second month for a week at a time, downtime also really affects how much sleep I get during that week. So perfect timing to read a book about pragmatic architecture.

System downtime can both cost huge amounts of money and seriously damage the reputation of a company. The incident that springs to mind when I think about damage to a company’s reputation is the Blackberry three day outage in 2011. It was the beginning of the end for a previously very successful company. Release It! is all about building a pragmatic architecture and infrastructure to avoid one of these doomsday scenarios for your company.

The book is organised into four different parts and roars into action with the first part: Stability or how to keep your system up 24 hours a day.

The Exception That Grounded An Airline

The author Michael Nygard has worked on several large, distributed systems and uses his war stories to illustrate anti-patterns.

The first story is of a check-in system for an airline that crashes early in the morning – peak time for check-ins. First, he describes the process of figuring out which servers were the problem (they had no errors in their logs). They then took the decision to restart them all. This restart took three hours and caused so many flight delays that it made the TV news. The post-mortem involved thread dumps and decompiling production binaries (there was bad blood between the developers and operations so they weren’t too keen to give him the source code). The bug turned out to be an uncaught sql exception that only ever got triggered during a database failover.

I love the this way of illustrating technical concepts. The stories are great and make the book a very easy read despite it being quite technical. And the postmortems after the incident are like CSI for programmers.

The lesson learned in this case was that it is unlikely that you can catch all the bugs but that you CAN stop them spreading and taking down the whole system. This bug was a simple bug but one unlikely to be ever caught in testing. Who tests that code works during database failovers and has that setup in their test environment?

However, by failing fast and using patterns like timeouts or circuit breakers you can stop the problem spreading through the whole system and crashing everything.

Cascading Failures vs Fail Fast

Circuit-breaker at High Voltage test bay, BerlinAfter setting the scene, the author then introduces a group of Stability anti-patterns. They have great names like Cascading Failures and Attacks of Self-Denial. These anti-patterns represent all the ways you can design an unstable system.

To counteract these anti-patterns are a group of Stability patterns.These include some very useful patterns like the practice of using timeouts so that calls don’t hang forever. Fail Fast counteracts the Cascading Failure anti-pattern. Instead of an error being propagated from subsystem to subsystem crashing each one, the idea is to fail as fast as you can so that only the subsystem where the error occurred is affected.

The most interesting pattern for me was the Circuit Breaker pattern. I haven’t used it yet in a production system but it got me looking at a .NET implementation called Polly.

Anyone else used it in a production system?

Capacity

The second part of the book introduces the capacity theme with a story about the launch of a retailer’s new website. They deployed the site at 9 a.m. and by 9:30 a.m. it had crashed and burned. Total nightmare. One of the sysadmins had a 36 hour shift setting up and deploying new app servers.

Again the author introduces a group of anti-patterns to watch out for and some patterns to counteract them. This time the anti-patterns are more obvious and are mostly some sort of waste of resources. They can be seemingly insignificant anti-patterns like not creating indexes for lookup columns in a database table, session times that are too long (a session in memory is using valuable memory), sending too much data over the wire or cookies that are too large. For a small system these are not a problem but if you multiple them with thousands of concurrent users then a lot more servers will be needed.

Michael Nygard makes a very good point here that most developers have less than 10 years of experience and most systems that are built do not have capacity problems; the combination of these two statistics means that very few developers have experience of working on large systems. This also means that the same mistakes are repeatedly made when designing large systems.

I personally still have a lot to learn here. I will be digging deeper into load testing and some of the more technical details of how our servers are configured. It’s easy to forget about how the code that I write affects the production environment.

General Design Issues

This part of the book is a collection of disparate design issues that can be important for large systems. Networking, security, load balancing, SLA’s, building a QA environment and other topics are covered here. Interesting stuff if not as compelling as the first two parts of the book.

Operations

The last part of the book kicks off with another great war story about a site suffering performance problems on Black Friday (the day after Thanksgiving in the US when every shop in the country has a sale). This time the problem was caused by an external system run by another company that had taken down two of its four servers for maintenance (on the busiest day of the year!). They were able to save the situation by reconfiguring a component not to call the external servers. This was possible due to good component design that allowed them to just restart the component instead of the server (it would have taken 6 hours to restart all the servers). The site was back working within a few minutes.

Transparency

So how did they know so fast what was wrong in this case? They had built the system to be transparent. Most systems are black boxes that do not allow any sort of insight into how healthy it is. To make a system transparent it is has to be designed to reveal its current state and a monitoring infrastructure is needed to create trends that can be easily analyzed by a human.

For example, at Tradera/eBay Sweden for our frontend servers we can see the number of errors per minute, page response time (both client and server side) and number of 404 and 500 HTTP response per minute (as well as lots of other measurements). For each server we can see memory usage, number of connections and a host of other performance indicators. By just glancing at our dashboard for live statistics I can see if we are suffering any performance issues on the frontend. Without this help it would be extremely difficult to keep the system running when things go wrong. As you can see this theme is close to my heart!

I did get a bit lost in the technical details of this section however. It is a really interesting theme but the technical details are either JVM-specific or have not aged well. A lot has happened in this space since the book was written in 2007.

Verdict

If you work with a system that is deployed on more than ten servers then this book is a must-read. There is a difference between just writing code and keeping a system up 24 hours a day. If you burn for Devops or are on-call regularly or care about creating stable systems then this book is for you.

 

 

Review of The Art of Unit Testing

The Art of Unit Testing With Examples in .NET by Roy Osherove, published by Manning. ISBN: 1933988274

Introduction

The Art of Unit Testing aims to teach developers how to write maintainable, readable and trustworthy unit tests. The author Roy Osherove has recently moved over to the Ruby community but is well-known in the .NET world for his TDD courses and TDD katas. He previously worked at TypeMock as Chief Architect so he lived and breathed unit tests for a few years.

This book is not what I had expected after reading all the online reviews. It is most definitely a beginner’s book, a book for someone just starting out with TDD. The first two books I read when getting started with TDD were Working Effectively with Legacy Code by Michael Feathers and Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat Pryce. The Art of Unit Testing refers heavily to them and I would describe it as an introduction to unit testing. I noticed that The Art of Unit Testing placed above them on Jurgen Appelo’s Top 100 Agile Books list and I do not understand that to be honest. It would have been the perfect text for me during the first month of beginning TDD but feels mostly redundant after reading the two afore mentioned books.

The Meat of the Book

The Art of Unit Testing is well-written and easy to read, I flew through it in a few sessions. The first three chapters are a very basic introduction to unit testing. It gets a bit more interesting after that with nice, clear explanations of stubbing and mocking in chapters 4 to 6. Chapter 6 also contains a section on creating a test API that I really liked. It is the section of the book that I am most likely to reread later. Chapter 7 goes a bit deeper into writing tests and is the most interesting part of the book. It manages to gather together most of the good tips on unit testing that I have seen before e.g. enforcing test isolation, practising DRY even in your tests, keep setup code simple, avoiding overspecification. Chapter 8 is non-technical and is about getting your organisation to start unit testing. The rest of the book lists the various unit testing tools and frameworks available as well as the classic TDD books that you should read.

The parts that could be better

The lists of tools has not dated too well. The book is written in 2009 and a lot has changed in two years. Mocking frameworks like NSubstitute and FakeItEasy are not included and Rhino Mocks gets a lot of attention. The newer BDD tools like Specflow are also not included. Those sections will be near worthless in a couple of years time.

The last chapter on working with legacy code is very skimpy for a subject that is both huge and difficult. And generally, a lot more could have been written about unit testing. Nothing about Object Mothers or Test Builders (check out this article from Los Techies or read Growing Object-Oriented Software, Guided by Tests). Not much on GUI testing with Selenium or integration testing the database or the BDD-style of testing. When I write code I do not really separate writing production code and writing unit tests, the two go hand in hand. And this makes the book less valuable than a good book on TDD even if I can understand why the author chose to do so. There are a lot of .NET developers out there who know nothing about unit testing or TDD and something like GOOS would scare off most of them.

The formatting of the code samples is dreadful, the alignment is off in almost half of them. You can still read and understand them but it feels low quality. I am reading another Manning book at the moment and its formatting is fine so I cannot understand how that got in to the final version.

And the verdict is…

I would recommend The Art of Unit Testing for anyone getting started with writing unit tests but not if you have been doing TDD for a while and have already read some of the recommended texts on TDD. It’s a nice, smooth read and very easy to absorb but it is only about unit testing and not TDD (a minus for me) and there is almost nothing about writing testable code, a huge part of testing in my opinion.

Review of The Nomadic Developer

Introduction

The Nomadic Developer by Aaron Erickson, published by Addison-Wesley. ISBN: 0321606396

The Nomadic Developer’s goal is to be a guidebook to working and thriving as a consultant. The author Aaron Erickson is a principal consultant at well-known ThoughtWorks and judging by his blog and Twitter output knows what he is talking about. I bought this book together with several other technical books but this was the first one I read. I felt I needed to balance my technical learning with some consulting skills.

The Nomadic Developer covers almost every part of being a consultant; everything from a technology consulting firm’s inner workings to personal advice about your appearance. For me, this structure means that the book is fairly disjointed and that the different sections have almost no connection except the consultancy theme. I also found that some of the themes are covered in depth and make interesting reading but that other sections are considerably weaker and only covered in a pretty shallow fashion. Amongst the stronger sections are the first few chapters that examine the inner workings of consultancy companies and consulting as well as some of the later chapters about breaking into the consultancy business and building a career path.

A Deep Dive into the Consulting Business

The first part of The Nomadic Developer is an introduction to technology consulting and examines why the consulting business exists and why you might want to work as a consultant. It’s a solid start to the book and I was drawn in straight away.

The second chapter presents “The Seven Deadly Firms” which are the seven types of consulting firms that you want to avoid. I found this chapter to be a bit weaker than the first chapter and slightly gimmicky. As the author says himself these seven types only cover 10% or less of all consulting firms out there but on the other hand it enables us to able to label these companies. Body shop is a well known term but Personality Cult Consulting and “Push the SKU” Consulting are new to me and pretty funny as well.

The third chapter does a deep dive into how technology consulting firms work and is one of the strongest parts of the book. These are the type of details that I did not know too much about before I became a consultant and am only starting to become of aware. It is quite different to working in an IT department or at a product company. The two pipelines of consulting, Sales and Recruiting are examined in detail and, as these pipelines are fairly new to me, this made fascinating reading.

Chapter four present the ten traits that technology consulting firms look for when hiring. This chapter read like one of those top ten blog posts that I usually skip read or just skip.

But straight after one of the weaker sections in the book follows a really strong chapter, one that I will reread a couple of times. It is all about interviewing your employer before taking the job. All the questions you need to ask about business, sales, delivery and community involvement not just the technical stuff that the Joel Test covers. I really wish I had read this before I started working as a consultant last year. The really tough questions that I would have liked to ask then but could not think of are right here.

Surviving and Thriving

I expected this section to be the main part of the book. All the chapters leading up to this section are preparation for the consulting life. So now I was all geared up to learn how to survive as a consultant and then how to be a great consultant.

Paradoxically, these chapters are the ones that got me thinking most about where I am headed with my career but also the ones that left me most disappointed. For example, the Thriving chapter touches on a lot of important topics but it just doesn’t go deep enough for my liking. In my opinion, a couple of the other chapters could have been cut and this chapter extended. I had hoped that the Surviving and Thriving as a consultant sections would give me more insights into such problems as handling difficult customers or how to be agile in non-agile environments. Instead a couple of topics like Building Your Brand and Learning to Think “Win-Win” are quickly skimmed through.

The author does point out some truths about the consulting industry which are important during a bad economic cycle. One of these is that rainmakers rarely get fired, something easily forgotten/ignored by a developer too caught up in the technical side of consulting. But it is mostly common sense or something that a new consultant will learn about during the first months on the job.

There is a chapter about avoiding career-limiting moves which uses the theme of the seven deadly sins that did not impress me too much. A mixture of common sense that applies to every job and moralizing that did not teach me anything and that I felt uncomfortable reading. I consider the section about dating co-workers and that it is a bad idea to surf porn at work to be totally unnecessary.

Conclusion

I am in two minds about whether to recommend this book or not.

On the negative side, I did not gain any earth shattering insights into the consulting world. There is way too much information which is either repetition or common sense. The book is pitched to an American audience and whole sections about avoiding getting fired did not really click with me (as a European). Some of the personal advice felt like moralizing to me and rubbed me up the wrong way. All calculations are done in dollars and the figures are based on American wages and costs which are so different from the European levels that it was difficult to translate them to my job and the company I work at.

But on the positive side this book has changed my behavior and how I look at my future career. And these are characteristics of the best books that I have read. Even if I learnt nothing new about potential career paths or building a personal brand, it still hammered home to me that I should focus more on these and not just on improving my technical skills. I have been thinking about this for the last two years but it crystallized into clear actions after reading The Nomadic Developer.

The parts about the inner workings of a Technology Consulting Firm were also very enlightening and went deep into the business side of consulting which would not be an area I know too much about.

So in conclusion, if you are a consultant and want to be a much better consultant then I do not think this book will teach you much. But if you want some tips on how to be better at interviews and how to build your career to be a successful consultant then this book will get you thinking.