“Agile Architecture”: Is Your Product’s Architecture Simply a Reflection of its Release Schedule?

After recently reading George Fairbanks’ Just Enough Software Architecture and obtaining 90% in my TOGAF 9.1 Foundation Certification exam in Enterprise Architecture, I’ve naturally been thinking about Software Architecture a lot. The other day a friend and I were talking about a typical scenario, involving large scale products with multiple sprint teams on a tight delivery schedule. I will over-simplify and over-generalize it here:

The Scenario

Let’s say you’re building a product which requires a feature that a snapshot can be exported to a PNG file. This is typically what would happen:

  • Product Owner: “As a user, I want to export snapshot to PNG, so that I can email it as an attachment to the Finance Department”
  • Story is added to Product Backlog, prioritized during the Sprint Grooming and estimated in the Sprint Planning
  • Developer Dinesh starts working on it, he adds the Export to PNG feature, it is demo’d at the end of the Sprint and everyone is happy

The feature and product iteration is shipped, customers start using it, the product team goes back to juggling new features and bugs. Priorities and sprint teams change. Somewhere down the line, another requirement comes up:

  • Product Owner: “As a user, I want to export snapshot to PDF, so that I can archive it in the Document Management system” (Don’t ask why the two systems can’t work with the same format).
  • Story is added, prioritized and estimated. This time though, Dinesh isn’t around (he’s either on vacation, promoted, no longer in the company or simply more focused on another aspect of the product)
  • Developer Gilfoyle starts working on this feature. From this point on, 3 things can happen:
    • Rearchitecture: An experienced developer or architect recognizes the potential of code reuse/refactor/rearchitecture between the two implementations. A common interface emerges, common functionality is moved up and specific functionality is moved to concrete classes. Note that:
      • Code is refactored
      • Some previously existing code is even thrown away
    • Technical Debt is accrued: Gilfoyle has limited time to implement the new feature; on the UI side the “Export to…” menu selections show up users would expect, but the internal implementation is disjoint. The code is not clean (but does the job), the Export functionality is not unified into a single interface and there is code duplication.
      •  In some of the stakeholders’ minds, code has been “reused” from the previous implementation, but in reality it has been copy-pasted (imitation being the best form of flattery towards Dinesh)
      • The team and PO might even recognize the Technical Debt and add it to the backlog, to be dealt with in the future at a lower priority (although that tends to lead towards a Bottomless Backlog)

In both cases, layers emerge in software over time as features are added and the product evolves. However, the first case has architectural layers and the second exhibits feature layers: In extreme cases, you might even be able to infer the Product Backlog by looking at the order in which features were added.

The Analysis

Some thoughts:

  • The Technical Debt Trap“: Doc Norton explained it very well at CraftConf 2016 (video link): (a) Most of what we have come to call “Technical Debt” is actually code cruft, and (b) “clean, testable code is a pre-requisite to being able to pay back Technical Debt”.
  • Show me the $: While a Product Owner may have limited influence over where budget is spent, a Product Manager may be in a better position to limit the long term negative effects of decisions made within the scope of individual sprints
  • One size does not fit all: Complex products or long development cycles, must be overseen by an experienced Architect or Development Lead, who advises the PO, PM and other decision makers
  • Stakeholders pay for the product, not the code: For stakeholders coming from a more traditional software engineering background, it may seem like throwing away (or refactoring) code is a questionable decision, because money was spent on producing it. This thinking must be erased by clarifying the benefits of a more robust and maintainable design.

The Solution (Maybe)

In summary: Features, as seen by the business, are not always the same as features, as seen by the developers. I’m tending to think the solution might be to maintain two backlogs: a business-oriented Product Backlog, and an architecture-oriented Technical Backlog. The latter would go into the details of how the business needs will be met at a technical level, driven by the objectives of overall cohesiveness, maintainability and constant reduction of Technical Debt. Here’s how I imagine it:2xBacklogs

Note that the technical effort to achieve a given amount of business features, is more than what it would be if the team worked off of a single backlog. This is simply because the Technical Backlog takes into account the additional effort of refactoring/rewriting, which is normally not covered in the typical Product Backlog.

I’m curious about what you think, please leave a comment below or get in touch on Twitter: @survivalcrziest.

(PS: Here’s a tip: VCS-based Software Analysis by Adam Tornhill)

Update: Prioritization of Items in the Technical Backlog

Based on some of the readers’ feedback, I would like to clarify how I intend the system to be used:

  1. The top of the Technical Backlog is prioritized / groomed based on (a) “Technical Value” and (b) budget and schedule constraints (“effort”).
    •  Technical Value here is defined as the value delivered by the technical solution towards meeting the needs of the [prioritized] Product Backlog.
  2. Items may be to the Technical Backlog not just based on the insertion of new features in the Product Backlog, but also based on technical reviews / retrospectives where it is identified that an architectural change is required (e.g. for purposes of maintainability or compliance).
  3. The items pushed to the bottom of the Technical Backlog thus represent Technical Debt.


Evangelizing Lean Product Development

I’ve written and tweeted before about how influential I think Eric Ries’ The Lean Startup has been for me. While we wait for Eric’s next book, The Startup Waywhich is more focused on bringing lean thinking to big companies (and by the way is being researched in a very interesting way — read about it on KickStarter), the timing was right for me to assimilate my thoughts and share them with a group of colleagues.

Lean Prezi pub

While I’m still waiting for the feedback to pour in for my 3-part course, I’m very happy to have had the opportunity to share with others that “Lean” is not just some black art practiced in car manufacturing plants, and “Entrepreneurship” can take root even in large, established organizations. Now, more than ever, we are all living (and trying to thrive) in “conditions of extreme uncertainty”.

I hope I have been able to get across some of the thinking behind the concepts covered in Eric’s book: the Build-Measure-Learn feedback loop, the Minimum Viable Product, Innovation Accounting, Vanity Metrics, practical techniques like Cohort Analysis & Five Whys and finally the most relevant: The Innovation Sandbox.

I also threw in a bit of George Fairbanks’ Just Enough Software Architecturewhich addresses “engineering risk”, as opposed to “project management risk”. And of course, lessons learned from my enlightening weekend at Lean Startup Machine a couple of years ago.

The coming days will tell if the course was useful and comprehensible enough for others to practically apply some of the principles to their projects. I hope that over time we can build a community of like-minded lean practitioners within the organization, acting as enablers of sustainable innovation. #WasteNot.

Software Design Simplified

I was remotely reviewing a new team member’s code:

It wasn’t that bad, but I wrote him a document suggesting some general design principles. I’m repeating some of them here, because they seemed to be the sort of thing that I would like to refer to often.

  1. For the typical application, start with the 3 basic layers: Front-end (GUI), Back-end (Database) and Business Logic. Give them their own classes within separate projects in the solution.
  • Ensure there are no cyclic dependencies
  1. Establish well-defined interfaces between the layers.
  • Internal implementation within a layer should be modifiable, with minimal (ideally zero) impact on other layers
  • Control should flow cleanly between layers, not back & forth
  1. Don’t duplicate code (or data structures) amongst layers (DRY Principle)

  2. At the highest level, each class should map to a real-world object that you intend to manipulate.

  • Then start breaking down the functionalities and add more classes at the lower levels
  • Try as much as possible to adhere to Object Oriented Principles, and at the very least to the one-class-one-responsibility principle (SOLID) (Annotation: I know it sounds obvious, but you’d be surprised what passes for “object oriented programming” these days)
  1. Don’t try to cover all the possible use cases up-front, instead aim for a design that has a solid foundation, on top which features can be added iteratively (YAGNI Principle)

  2. The earlier you put in place a flexible logging mechanism (whether your own or reused), the happier you’ll be. No application is too small to benefit from logs. Besides debug info, they can also serve a source of useful information that the user need not be burdened with under normal circumstances.

  3. Prioritization: Normally I would proceed in the order:

  • Functionality, Reliability
  • Usability
  • Robustness, Security
  • Efficiency
  • Maintainability, Portability
  1. Tools:
  • Mindmaps for capturing Requirements & Design (Reference)
  • Dia for flowcharts  (Annotation: Yeah, I’m old skool)
  • StarUML for UML Diagrams

XML Document Design Considerations

Resources on design considerations of XML documents: