Craft Conf 2015, Day 2

I had the privilege of attending the second year of CRAFT, a tech conference in Budapest focused on software craftsmanship. The last year (which was the first time it was held) had completely blown my mind. A year later I still keep referring back to the talks and haven’t finished fully absorbing them and putting all those inspiring ideas into practice.

IMG_3467

In short…

Craft Conf 2014 was better. The speakers came from a more diverse background, the talks spanned a multitude of unrelated topics and I remember it being very, very hard to choose from talks happening in parallel. Each minute spent there was a revelation.

This year, though, many of the talks seemed to be plug for a company or a product, in disguise. Certainly there were brilliant takeaways, but not at the same scale as the previous year.

In my opinion, 2014 was also held in a better venue, although the 2015 venue was outstanding too, as far as tech conference venues go. But the rooms were too far spread out (the map was inaccurate), the acoustics were bad everywhere except the Main Room and unlike 2014, the WiFi was not flawless. Lastly, there were far fewer food choices, longer queues, no bottled water (even for the speakers) and therefore a lot of glasses clanking.

On the positive side, the schedule was followed down to the minute, the live video streaming was smooth and considering the scale of the event (1300 attendees), everything was beautifully organized. I’m not complaining – it’s just that the first CRAFT had set a pretty high standard.

(Video and Slides links will be updated by next weekend, when they become available)

Agile Engineering in a Safety-Critical World

By Nancy Van Schooenderwoert | Video | Slides

“Instead of freezing the ocean, learn to ride the waves” – Nancy’s talk was mainly about how our need for predictability for effective coordination is at odds with our need for fast learning to handle unknowns. She pointed out that in the agile context, “Architecture is any design decision that you cannot easily change”.

There was the customary reference to WikiSpeed to dispel the myth that hardware changes can’t fit within 2-4 week iterations. And an interesting one to a paper called TIR45 from AAMI: Guidance on the use of AGILE practices in the development of medical device software.

Coding Culture

By Sven Peters | Video | Slides

Sven’s talk was both informative and inspiring. Some of the key takeaways:

  • Innovation needs time
  • Stop and celebrate wins, however small they may be
  • Balance your passion for code with your passion for customers
  • Turn your passion into product
  • Value trust, autonomy and transparency (Atlassian achieves this by using chat over other communication means)
  • Products come and go, culture stays

Take a look at Atlassian’s Mood App and Stash Reviewer Suggester.

Building Reliable Distributed Data Systems

By Jeremy Edberg | Video | Slides

This one was good, until we went deep diving into the NetFlix Simian Army, which was also good but could have been summarized in just one slide. One thing that stood out from Jeremy’s advice was to “build for three”, because if you can overcome problems there then the solution can be [more] easily scaled up to n.

Don’t forget to check out NetFlix Open Source Software Center.

Oh! You Pretty Tools

By Andrew Bayer | Video | Slides

Andrew gave an interesting talk about the role of internal tools and their developers in the organization, covering both the pros and the cons. For example while making the build or buy decision, consider the fact that people are more expensive than software. And some thoughtful insights, like how Integration Tests can double-up as the roadmap to your tool’s usage. He also revealed that Cloudera runs ~2000 Jenkins CI builds every day(!)

The rest of it was basically about, and lessons learned from, CloudCat.

Testing and Integration (The Remix)

By Ines Sombra | Video | Slides

Ines entertainingly summarized everything we know so far, and topped it off with new insights for a good measure.  She emphasized:

  • The importance of lightweight short-lived branches so that CI is not overlooked
  • The more likely a test is to fail, the sooner you should run it
  • The testing of provisioning systems, such as Chef Recipes, too
  • How test setup time and parallelization are the key factors in minimizing the testing cycle time

She recommended this talk about the Google Build System.

Her punchline was that CI is a predictor of professional maturity at the organizational and individual level, and she ended with a “rantifesto” about building a culture of quality.

Beyond Features: Rethinking Agile Planning and Tracking

By Dan North | Video | Slides

From Cutting to Curing: Dan presented the powerful and inspiring idea that maybe software engineering is more like surgery than the civil engineering principles that we currently use to manage it. Agile methodologies essentially optimize for predictability, and this not necessarily a good thing. He mused on how a 2-week sprint is just enough time for a mini-waterfall, and thus we are all basically whitewater rafting.

After reviewing where the Agile Manifesto has brought us, he set an ambitious new goal to sustainably minimize the lead time to business impact. 

He ended with:

  • The role of Features, Delivery and Kaizen
  • Schedule, Measure, Track, Showcase
  • How Value Stream Mapping can reveal surprises like typically a piece of work spends upto 90% of it’s time waiting for dependencies

How To Save Innovation From Itself

By Alf Rehn | Video | Slides

For me, Alf’s talk was the highlight of the event. It was so good and so inspiring that I won’t even summarize it here. Go watch it!

______________________________________________

The day ended with a party thrown by EPAM, which included free beer, a DJ-saxophone duo and a surprise flashmob.

You may also want to read my review of Day 3 of Craft Conf 2015.

 

 

 

 

 

Takeaways from Craft Conference 2014, Budapest – Day 3

Continued from Day 2, here are the talks from Day 3 in order of my personal preference (and relevance) which may differ from yours:

Complex Projects aren’t planable but controllable

by Jutta Eckstein (Slides | Video)

Sadly, Jutta’s slides aren’t available online but the talk was packed with solid advice for Project Managers. Some of them were:

  • Our predictions are usually based on a coherent story, not a complex, dynamically changing reality
  • Targets should be ambitions, not absolute
  • Focus on the value gained, rather estimates of the effort required
  • Having an  annual budget is like having a bank that opens for business only once a year
  • Annual budgets are not optimal because they are never underutilized: the excess (if any) is still used up, never returned back
  • Consider [event-based] rolling budgets, rolling plans and rolling control
  • Check value and progress regularly
  • Don’t just trust the experts, seek feedback from diverse groups
  • Have different planning strategies for Roadmap (themes only), Release (features based on value & velocity) and Iteration (Stories based on value (+estimate) & velocity)

Recommended reading:

Architecture War Stories

by Stefan Tilkov (Slides | Video)

Probably the most amusing talk of the event. Stefan shared some hilarious real-life examples of architectural disasters… some of them still very much in use (of course no names were revealed). His advice was to go back to the basics:

  • Make data free of code dependencies
  • If it makes you want to pull your eyes out, maybe you shouldn’t be doing it
  • Better ask for forgiveness than permission
  • Development environment is not the same as production environment
  • Feedback, reflection, iteration

Responsibly maximizing craftsmanship in software engineering

by Theo Schlossnagle (Slides | Video)

Theo revisited the basic problems plaguing software development today and went on to share some of his experiences at Circonus. One of the tips was to treat all software you write as open-source within the organization, even if you don’t plan to release the source outside. Other highlights:

  • Turtles all the way down: Software at scale is tied together with loose string & hope
  • We use human language to describe software specifications, and this can be interpreted by different people in different ways (like poetry)
  • Technical debt is non-linear; large monolithic components are more likely to fail because they are hard to maintain
  • Reusability is good, but use the right tools for the right purpose. Accept that that right tool may not yet exist and may need to be written
  • Diversity is an emergent property of scale
  • Thou shalt be judged by your API

For software developers, the message was loud and clear:

kPhone_IMG_0951

Data-Driven Software Engineering

by Jevgeni Kabanov (Slides | Video)

Jevgeni went thorough the effort of analyzing 1000+ projects on Productivity, Quality and Predictability. Sadly the slides aren’t available, but the raw data they were based on is. You can still watch the video and grasp the most important points, such as the fact that the best software projects are delivered 80% of the time on schedule and still have 25% critical issues. Or that automated tests improve quality by 26%. There were also secondary interpretations such as code reviews improve architecture, not just code quality.

Jevgeni emphasized on the importance of measure & experiment over simply “doing agile”. Some suggestions for measurement were:

  • Deadline misses
  • Scope changes
  • Blockers after release
  • User satisfaction

The presentation became a bit controversial later but as Jevgeni said: “People, chill – I gathered data and presented my analysis – feel free to take the data and do a better job”. And I think this talk had the most creative closing slide ever 😉

Without Present or Past: How to Think Distributed – The Hard Way

by Endre Sandor Varga (Slides | Video)
By far the most profound and philosophical talks at Craft. If you have an interest in Distributed Systems, AI or Information Architecture, I would strongly recommend watching the video. The core concept was based on applying epsitemic thinking to systems that are distributed, concurrent, able to fail independently, and communicate over a lossy medium with non-deterministic communication delay. Endre touched upon:
  • Two Armies Problem
  • The final important message
  • Difference between actual state and observed state
  • The Omnipotent Observer (doesn’t exist, because state is always queried)
  • Global introspection and self-awareness
  • Cone of the past
  • The present is volatile and subjective
  • Going from someone knows -> someone knows everyone knows -> everyone knows everyone knows

Endre ended with this advice:

  • Have a properly defined failure model
  • Never assume reliable communication
  • Never assume common knowledge

Functional Reactive Programming in Elm and JS

by Evan Czaplicki (Slides | Video)

Evan is the designer of the Elm language and his enthusiasm is infectious. Over the course of his talk, he built and demonstrated a simplified (yet slick) Mario game complete with physics and reaction to keyboard inputs. Elm is a Functional Reactive Programming language for web browser based GUIs and this game was a fine demonstration of concepts that Bodil Stokke and Jonas Boner touched upon in their respective talks.

There is no doubt that Elm is a game-changer. Here is another example of traffic simulation.

Further reading:

Find the Right Abstraction Level for Your Tests

by Gerard Meszaros (Slides | Video)

If you’ve ever been haunted by the question of what level of testing is enough (and who hasn’t?) then this presentation was for you. The key message was to think in terms of not what you can add, but what you can leave out of the test: “if it isn’t essential to conveying the essence of the behavior, it is essential to not include it”. Over 39 slides, Gerard illustrated this step-by-step with one, continuous, easy-to-follow example.

Software Psychology: The Art of Listening to Code

by Bjorn Freeman-Benson (Slides | Video)

Bjorn talked about the concept of code screams: behavioral indications of a deeper problem in the system. For me the key takeaway was that you should continuously monitor your processes as well as your systems in production (e.g. by gathering usage statistics) and fix the root cause when a problem is found.

Others

Here are some talks that I missed, but which received a lot of positive feedback. Thanks to ustream.tv they are available online. Also check out #CraftConf on Eventifier.

Jackstones: the journey to mastery

by Dan North (Slides | Video)

McDonalds, Six Sigma, and Offshore Outsourcing: Unexpected Sources of Insight

by Chad Fowler (Slides | Video)

Testing the Hard Stuff and Staying Sane

by John Hughes (Slides | Video)

The Better Parts

by Douglas Crockford (Slides | Video)

Functional Examples from Category Theory

by Alissa Pajer (Slides | Video)

 

Takeaways from Craft Conference 2014, Budapest – Day 2

I had the privilege of attending the speaker sessions of Craft Conference last week, the central theme of which was software craftsmanship. There were many inspiring talks and so was the venue. Think of it as TED for software developers. The icing on the cake was free beer, complimentary Palinka, unlimited coffee and a blues band. 20% of the speakers were women (but only 4% of the attendees) and ~350 of the 900+ attendees were from abroad. The event was virtually flawless. The usual systems that breakdown at scale: WiFi, food and toilets, all just worked. Plus there were small thoughtful touches such as English translations of useful Hungarian phrases on the attendee badge. Everything about the conference was impressive, including the raffle prizes which included a R/C drone!

kPhone_IMG_0936

Other interesting features were the use of Sli.do to manage audience questions in real-time, and a live-projected twitter feed. I had the opportunity to interact with practicing or aspiring software crafts[wo]men from Ukraine, Japan (ok, technically SF), Netherlands, Sweden, North America, UK and… India!

I’ve tried to distil out the summary of talks that interested me. They are ordered by my preference, not the order in which talks were actually conducted. These are my interpretations and my views, so please bear in mind that they could be wrong or biased. There were 3 tracks (parallel talks) over 2 days, so essentially I attended only about 1/3rd of the total. Day 2 was the first day of the speaker talks… I didn’t attend day 1, which was workshops.

I strongly recommend checking out the agenda and viewing the talks that interest you online, you might find some that I didn’t attend but are of direct interest to you. To me the top 3 recurring themes of the conference seemed to be:

  1. State management in complex and distributed systems
  2. Better automated testing & TDD
  3. The abuse of agile (in development and in project management)

How I Learned to Stop Worrying and Love Flexible Scope

by Gojko Adzic (Slides | Video)

Probably one of the most-loved and honest talks at Craft. The storytelling was simply mind-blowing (an example of agile from 1628 AD, Ducati’s experience with the Second System Syndrome, threats to competitive advantage from Google & the Russians) and so was the message. I would highly recommend watching the video, Gojko is an outstanding speaker. The highlights were:

  • The most common software development methodology these days is WaterScrumFall, in which all the essential planning is done upfront by management and the development is done in a [predictable] number of sprints
  • This is because the concept of agile is not attractive to management, unless they truly believe in keeping scope flexible. Therefore agile generally remains underutilized.
  • Agile is not just about continuous delivery, but also continuously reacting to local, temporal and human factors
  • Project plans and roadmaps should not be linear, but literally a “map of roads”: multiple options with selection criteria (like a GPS). A roadmap with a pre-decided outcome is not a roadmap, but a road in a tunnel.
  • Try new things, at a survivable scale and select the ones that work (Throwing away bad code is a way to reduce your technical debt)
  • Don’t just ship software, make an impact
  • On the topic of outsourcing: Usually the objective is to minimize costs, so the focus is not on excellence or flexible scope
  • On User Stories:
    • Are a way to delegate details
    • Avoid translating the product roadmap into a set of JIRA stories, instead consider hierarchical backlogs
    • Add a victory condition to your user stories, which is related to changing user behavior rather than complying with the existing behavior (e.g. Monitor inventory *faster*)
    • A good user story is a survivable experiment

Recommended reading:

Agility and the essence of software architecture

by Simon Brown (Slides | Video)

This was also one of my favorite talks. I like to think of it as Minimum Viable Architecture for developers. Simon is an inspiring speaker, more so because he eats his own dog food. The premise was simple:

  • Agile delivery does not imply agile architecture
  • Agile development still requires upfront thinking to define the overall architecture
  • The team must have a shared, consistent vision of the significant structural elements of the product
    • With this pragmatic tradeoffs can be achieved: e.g. a monolithic deployment container, containing microservices
  • UML was supposed to solve this problem, but it is poorly understood, not widely adopted and has its own overheads

The solution: NoUML! Abstraction is more important than notations. A team can define their own legend for these abstractions. Design diagrams are supposed to reduce complexity (through abstraction), not increase it. The 3 things that the team needs to have a common understanding of are:

  • Structure
  • Risks
  • Vision

In real life, we rarely have 1:1 mapping between design diagrams and code. In general, a hierarchical C4 architecture diagram can solve this problem:

  • Context
  • Container
  • Components
  • And optionally, Classes

And yes, while good architecture is a shared responsibility, it is important that only one person (or a small group of people) are responsible for maintaining the overall architecture definition.

Bring Software Architecture Back!

Recommended reading:

Getting Things Done at Scale

by Amber Case (Slides | Video)

This was a talk I could relate to a lot, because of GTD and the differences in corporate cultures of large, old organizations vs smaller, newer ones. Amber is a TED speaker, and it clearly shows. She shared her experiences during (and after) the acquisition of Geoloqi by ESRI.

Only 1 of 50 people she spoke to for advice had a happy acquisition experience, and the main reasons were:

Crippling management/overhead to get simple things done Culture clash
Founder flight Jealousy/blocking from parent company employees
Lack of detailed transition plan Sprinters vs Marathon Runners
Loss of passion for original product Loss of respect and cross-compromise

The solutions discussed were:

  • Pre-negotiate, don’t be vague. Predetermine your outcome.
  • Learn the local language (technical terms, tools, company culture)
  • Win friends to influence people. Develop trust.
  • “Beta test” people via small projects. Best code wins.
  • Scale teams down from unmanageable numbers to 5-6 “doers”
  • Communicate. Respect. Give first.
  • Small revolutions
  • Distribute stress

An interesting concept was the creative use of IRC bots, e.g. to send out a daily email summary of accomplishments (!done). Towards the end, she also shared an effective way of “organic hiring”: turning contractors into full-time employees.

It’s never too late to fight your legacy!

by Mate Nadansi (Slides | Video)

Mate delivered a very strong and reassuring message: Legacy code isn’t bad, just old or over-iterated… and, with some sensible planning, foresight, and many iterations of hard work, it can be replaced by more modern code. He explained how they achieved exactly that at ustream.tv. The presentation requires some familiarity with web programming but it would still benefit anyone working with legacy code… because maintaining legacy code builds character. 😎

Programming, Only Better

by Bodil Stokke (Slides | Video)

The core message was about how the introduction of multiple [mutable] states makes programs unmanageably complex. Topics included:

  • Referential Transparency
  • Representation of state using numbers instead of objects to it remains immutable
  • “Encapsulated state” is still state
  • Effect on concurrency
  • Additional complexity added by control structures in contemporary program

The presentation itself was a work of art and for me the highlight was that Bodil was editing & running code from within the slides!

Recommended reading:

  • Out of the Tar Pit, a 2006 paper about Functional Relational Programming by Ben Moseley and Peter Marks

Going Reactive: Event-Driven, Scalable, Resilient & Responsive Systems

by Jonas Boner (Slides | Video)

Jonas gave an inspiring and pragmatic talk about how the nature of, and expectations from, applications have changed dramatically over the years. The highlight was that not only did he distill them into the 4 attributes mentioned in the topic of the talk, but also shared valuable insights into how to practically achieve them from a technical standpoint. Some of the approaches discussed were:

  • Loose coupling
  • Never blocking
  • Asynchronous operations
  • Actors and Agents
  • Futures in Scala
  • Minimizing contention
  • Maximizing locality of reference
  • Single Writer Principle
  • The relation between scaling up and scaling out
  • Decoupling error handling from business logic
  • Bulkheading to prevent cascading failures
  • Maintaining consistent latency across blue skies conditions, high traffic and failure

Recommended reading:

What Makes a Good Development Process?

by Bruce Eckel (Slides | Video)

Bruce Eckel is an industry veteran an author who needs no introduction (I read his textbooks in high school). I highly recommend going over his slides, they are very comprehensive, thoughtfully put together and highly informative. He has already distilled years of experience into a few pages, and I don’t think I can add any more value except quoting the one thing that stood out for me: The cognitive load of carrying tensions prevents us from doing creative tasks well.

Acknowledging CAP at the Root – in the Domain Model

by Eric Evans (Slides | Video)

This talk dealt with a very specific type of problem (CAP = Consistency, Availability & Partition tolerance), and solving it using a domain-driven programming model. One of the interesting concepts was that of defining aggregates within a distributed system. Instead of trying to keep the entire system in a consistent state, the contents of each aggregate are guaranteed to be consistent (even if equally stale). There is only a single point of reference into an aggregate. Aggregates are contained within a bounded context and transactional consistency must not cross these boundaries. Domain events interpreted within this context cause state changes. Eventually overall system consistency can be achieved through synchronization according to a reasonable SLA.

Don’t forget to check out Day 3!

FilesDesk

2012

This started out as a side project between a couple of us at work to sharpen our C#/XML skills and generally do something challenging. A C#.NET Application + Service that enables browsing files on disk by Tag (as opposed to Path). Typical usage:

  • A Tag Cloud for your hard disk
  • Add/remove tag(s) to selected file(s)
  • List & Filter files by tag(s)
  • Monitor specified folders (ex: My Documents) for new/deleted/renamed files
  • Portable XML Database

Useful for Researchers, Writers and Obsessive-Compulsive File Hoarders.

DOPE: Distributed OPerating Environment

November 2003

I did this project with a friend around that time that Grid Computing was getting popular in the mainstream with SETI@Home (in its pre-BOINC avatar).

A Client-Server Java app that used Distributed Computing principles to:
a) Break down a given, computationally intensive task into chunks*
b) Distribute these chunks to clients on the network, favouring those with lower latency
c) Re-assemble the results returned by clients into a single unit
d) Re-send chunks for which result was not received within stipulated time

*Limitations apply: this was not a general-purpose implementation

“Different Algorithms in Java”

Recently ran into this matchless post at the Sun Developer Network (SDN) about the “different algorithms in Java”. Make sure you aren’t drinking anything while reading this 😉

SequoiaView and TreeMaps

SequoiaView uses the concept of squarified treemaps to present the contents of your computer hard disk in a novel way. Also see the website of their spin-off company MagnaView, which specializes in visualizing huge amounts of data.

Algorithms, Data Structures

Algorithms and Data Structures are the DNA of computers, they are what run the world that we live in today, and they will probably continue to run the world long after the human race wipes itself off the surface of this planet. Algorithms help us understand life and increasingly in the future will help us in improving it. When I was a child, I was taught that what a computer does is basically INPUT -> PROCESS -> OUTPUT. That is the core of computing: the INPUT and the OUTPUT are Data Structures and the PROCESS is the Algorithm.

Resources:

References:

  • Introduction to Algorithms (Second Edition), By Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest and Clifford Stein (MIT Press, 2001)
  • Data Structures Using C and C++ (Second Edition), By Yedidyah Langsam, Moshe J. Augenstein and Aaron M. Tenenbaum (Prentice Hall, 1995)