“Agile Architecture”: Is Your Product’s Architecture Simply a Reflection of its Release Schedule?

After recently reading George Fairbanks’ Just Enough Software Architecture and obtaining 90% in my TOGAF 9.1 Foundation Certification exam in Enterprise Architecture, I’ve naturally been thinking about Software Architecture a lot. The other day a friend and I were talking about a typical scenario, involving large scale products with multiple sprint teams on a tight delivery schedule. I will over-simplify and over-generalize it here:

The Scenario

Let’s say you’re building a product which requires a feature that a snapshot can be exported to a PNG file. This is typically what would happen:

  • Product Owner: “As a user, I want to export snapshot to PNG, so that I can email it as an attachment to the Finance Department”
  • Story is added to Product Backlog, prioritized during the Sprint Grooming and estimated in the Sprint Planning
  • Developer Dinesh starts working on it, he adds the Export to PNG feature, it is demo’d at the end of the Sprint and everyone is happy

The feature and product iteration is shipped, customers start using it, the product team goes back to juggling new features and bugs. Priorities and sprint teams change. Somewhere down the line, another requirement comes up:

  • Product Owner: “As a user, I want to export snapshot to PDF, so that I can archive it in the Document Management system” (Don’t ask why the two systems can’t work with the same format).
  • Story is added, prioritized and estimated. This time though, Dinesh isn’t around (he’s either on vacation, promoted, no longer in the company or simply more focused on another aspect of the product)
  • Developer Gilfoyle starts working on this feature. From this point on, 3 things can happen:
    • Rearchitecture: An experienced developer or architect recognizes the potential of code reuse/refactor/rearchitecture between the two implementations. A common interface emerges, common functionality is moved up and specific functionality is moved to concrete classes. Note that:
      • Code is refactored
      • Some previously existing code is even thrown away
    • Technical Debt is accrued: Gilfoyle has limited time to implement the new feature; on the UI side the “Export to…” menu selections show up users would expect, but the internal implementation is disjoint. The code is not clean (but does the job), the Export functionality is not unified into a single interface and there is code duplication.
      •  In some of the stakeholders’ minds, code has been “reused” from the previous implementation, but in reality it has been copy-pasted (imitation being the best form of flattery towards Dinesh)
      • The team and PO might even recognize the Technical Debt and add it to the backlog, to be dealt with in the future at a lower priority (although that tends to lead towards a Bottomless Backlog)

In both cases, layers emerge in software over time as features are added and the product evolves. However, the first case has architectural layers and the second exhibits feature layers: In extreme cases, you might even be able to infer the Product Backlog by looking at the order in which features were added.

The Analysis

Some thoughts:

  • The Technical Debt Trap“: Doc Norton explained it very well at CraftConf 2016 (video link): (a) Most of what we have come to call “Technical Debt” is actually code cruft, and (b) “clean, testable code is a pre-requisite to being able to pay back Technical Debt”.
  • Show me the $: While a Product Owner may have limited influence over where budget is spent, a Product Manager may be in a better position to limit the long term negative effects of decisions made within the scope of individual sprints
  • One size does not fit all: Complex products or long development cycles, must be overseen by an experienced Architect or Development Lead, who advises the PO, PM and other decision makers
  • Stakeholders pay for the product, not the code: For stakeholders coming from a more traditional software engineering background, it may seem like throwing away (or refactoring) code is a questionable decision, because money was spent on producing it. This thinking must be erased by clarifying the benefits of a more robust and maintainable design.

The Solution (Maybe)

In summary: Features, as seen by the business, are not always the same as features, as seen by the developers. I’m tending to think the solution might be to maintain two backlogs: a business-oriented Product Backlog, and an architecture-oriented Technical Backlog. The latter would go into the details of how the business needs will be met at a technical level, driven by the objectives of overall cohesiveness, maintainability and constant reduction of Technical Debt. Here’s how I imagine it:2xBacklogs

Note that the technical effort to achieve a given amount of business features, is more than what it would be if the team worked off of a single backlog. This is simply because the Technical Backlog takes into account the additional effort of refactoring/rewriting, which is normally not covered in the typical Product Backlog.

I’m curious about what you think, please leave a comment below or get in touch on Twitter: @survivalcrziest.

(PS: Here’s a tip: VCS-based Software Analysis by Adam Tornhill)

Update: Prioritization of Items in the Technical Backlog

Based on some of the readers’ feedback, I would like to clarify how I intend the system to be used:

  1. The top of the Technical Backlog is prioritized / groomed based on (a) “Technical Value” and (b) budget and schedule constraints (“effort”).
    •  Technical Value here is defined as the value delivered by the technical solution towards meeting the needs of the [prioritized] Product Backlog.
  2. Items may be to the Technical Backlog not just based on the insertion of new features in the Product Backlog, but also based on technical reviews / retrospectives where it is identified that an architectural change is required (e.g. for purposes of maintainability or compliance).
  3. The items pushed to the bottom of the Technical Backlog thus represent Technical Debt.

 

Advertisements

The Art of Listing

As I had posted here in 2007, I have been trying to make an effort towards paperless organization of my lists, most of which are ToDo items. The Palm device that I was originally attempting to use for this effort turned out to be a headache because of limitations on formats, storage capacity, speed, interoperability and expandability. I ended up giving it away to my cousin brother who is a student, to make his first attempt at getting organized 🙂 In the meantime, I picked up a Sony Ericsson P990i, which let me do a lot more, faster and more efficiently (Of course, that device is also fast approaching its event horizon). I found that I have so much going on in my head that often it was a pain to take out the phone, flip it open, navigate to “Tasks” or “Notes” and start typing. Going 100% paperless wasn’t working out too well, sometimes during this physical process I would lose track of my mental process (i.e. forget the idea or task that I wanted to note down). Over the years, I have arrived at the following hybrid approach, which helps me get things done effectively:

1. On my device, I maintain the following lists, in the following order, each of them almost like a Product Backlog:

1) This Week 5) Online – Stuff to do the next time I’m in front of a computer, like e-mailing somebody
2) Weekend 6) Projects – Not just software, even real-world projects like scale models
3) Next Week 7) This Quarter
4) This Month 8) This Year

Plus, the following “dynamic” lists:
a) Groceries – Since the stuff I buy every week/month is almost always the same, I just have a master list in which I keep moving things between “Pending” (unchecked) and “Completed” (checked) depending on what I run out of
b) Shopping – Other things to buy next time I’m out
c) Travel – Places to travel to on the weekends
d) Focus – 1 to 5 items I’m currently focusing on (e.g. “Get to work on time” 🙂 ), to keep reminding myself regularly

2. My phone lets me prioritize tasks within each list, on a scale of 1 – 3. Also, for example, within “This Month”, if “Pay Rent” has been completed, it gets checked into “Completed” and doesn’t get deleted. At the beginning on next month, I simply uncheck everything back into “Pending”.

3. I maintain a single sheet of pocket notebook-sized paper (more if I’m actively noting down ideas/tasks for an ongoing activity/project), akin to a Sprint Backlog, with the following:

Front Side Back Side
Today – Things to do today (mostly at work) This Week – Including weekend commitments
Calls – Phone calls to make + e-mails to send Home – Things to do when I get back from work

4. Every weekend I move stuff from the “Product Backlog” (long-term list of stuff on the phone) to the “Sprint Backlog” (short-term list of stuff on paper), and *wait for it* stuff gets done! I never use more than one sheet of pocket notebook-sized paper in a week, and this way I also always have paper handy to quickly note down stuff (on the margins). Finally, in case I ever lose/damage my phone (which is backed up every 2 weeks), I don’t lose the things I had planned for the week.

Am I going overboard? (After all, it’s just a glorified ToDo list.) I don’t think so. I find that by keeping things prioritized and focused this way:

1.  I manage to get a lot more done without worrying about what I’m forgetting to do.

2. I don’t lose track of things that I would eventually like to do, but don’t have the time for right now (or this week, or this month, …)

3. Moving the prioritization and organization out of my head helps me think clearer and focus 100% on the task at hand.

But it doesn’t end there. Over a period of time (and with a lot of self-imposed discipline, I must add), I have managed to harmonize the short-term (a.k.a. “sort it when you see it”) organization of things that I come across everyday. I do this by managing the following “tags” (often as Folders, in some cases even physical file folders) across my Inboxes, Browser Bookmarks, Hard Disks and scattered notes (including those on my phone):

  • BlogThis
  • ReadThis
  • WatchThis
  • DownloadThis
  • FollowUp

I visit these as and when I have the time and keep emptying them out. With the addition of lists (as notes) for Movies to watch, Music to get, Books to read and Scale Models to buy, my little universe of lists is complete!

Stuff that I learned along the way, though:

1. Hybrid is more practical than paperless.

2. We need a device (implant?) that can make a note when the wearer thinks of it (and where to put it). The interface & actions between thought and task noted are, well, so ’90s! (Note: Speech Recognition is also so ’90s)

3. It’s best to stick to simple formats like Text and CSV instead of proprietary ones (like Excel). Simpler formats are easily portable and retrievable in case of failure, and suffice for making lists. If your list seems to require a complicated format, well, simplify your list!

4. It may be a good idea to reuse Visiting Cards and such, but your handwriting needs to be tiny.

5. Evernote can probably help.

UPDATE: [2011-07-16] I have since migrated all my lists to my new BlackBerry Curve 9300.

UPDATE: [2012-03-09] I discovered Todoist, which is quite simply the Tao of using Computer Science to solve problems. Although using it means that my todo list is now in the cloud, something that I’m [still] not very comfortable with, I find it indispensable to manage long-term projects. I initially found it attractive due to its Outlook integration, which meant that I didn’t have to grapple with numerous tasks disguised as emails, but the app is constantly being improved with new features, like @labels that enable a task to be present in multiple lists. HTML5 support means my list is now available offline, and it syncs effortlessly across devices, including my BlackBerry. I occasionally take local backups with Todoist Backup.

UPDATE: [2012-09-15] With my mind emptied of the long term stuff now safe on Todoist, I have started relying more on my memory for day-to-day things. I’m also trying to do less and focus more on the important things (not to mention years of long hours have significantly shortened my “backlogs”), and try not to take on more than I can comfortably remember over the span of a few days at a time.

UPDATE: [2012-10-05] I have a new revived obsession with Whiteboards at home. I’m trying to keep it under check to avoid looking like too much of a mad scientist…

UPDATE: [2013-07-12] Since Todoist Backup no longer works and I don’t have the time to figure out what changed, I have upgraded to Todoist Premium to be able to make use of their backup service (amongst other cool features). I now exclusively use Todoist coupled with a one-page note on my phone, which has tasks for the day to week range, plus trivial items (< 2 min) that don’t need to go into Todoist. Also, the other day I found the phone I used to use before my Palm device: Nokia 6820.