An Exercise in Analytics: Using Microsoft SandDance to Visualize Trends at CraftConf

I’ve been lucky to attend every CraftConf Budapest since its inception in 2014.  It has always been a mind-expanding experience, with a healthy mix of established and newcomer speakers from the tech industry worldwide. Although Craft, as the name suggests, is focused on Software Craftsmanship, the specific topics of talks vary each year as industry trends fluctuate. I was interested in taking a deeper look at trends of technologies and paradigms as they become more or less popular over time, and to find those, if any, whose popularity has remained more or less constant.

While doing some research into data analytics for a project at work, I came across Microsoft SandDance. My interest in the CraftConf data was the perfect opportunity to teach myself SandDance (and some Python as well). So I put together a quick experiment: Using Martin’s excellent tutorial on Web Scraping with Python as a reference, I wrote  a Python script that scrapes CraftConf’s talk descriptions (using archives from 2014, 2015 and 2016) and produces a CSV file of the most frequently occurring words. Obviously standard English language words like pronouns, days of the week, and so on are ignored. What we’re left with is a list of top 100 words for each year, and their frequencies of occurrence, which can be visualized  in SandDance.

Although this approach is quick and more or less effective, the limitation is that it may not accurately reflect trends for phrases like “Agile Methodology” — the word frequency of “agile” may not be the same as that of “methodology”. But that’s something that can be worked on later. So although I would take this as a good indicator (which meets my purpose), I wouldn’t use the analysis outcomes as a serious reference.

What the Data Tells Us

Here are some interesting findings from the first pass (150 of the most frequent words selected, out of those 100 produced after filtering out common and punctuated words):

  • “Product” showed up 26 times in 2015 and nearly doubled to 51 in 2016 (a growing trend: no talk, some talk, twice the talk…)
  • “Functional” [programming] appeared 16 times in 2014, and not in the other 2 years (something that’s coming and going?)
  • Similarly, “Architecture” showed up respectively 15, 37 and 23 times (up and down)
  • “DevOps” was an equally hot trend in 2014 and 2015 but didn’t show up in 2016 (presumably because the hype is over)
  • “Microservices” appears 29 times in 2016, but didn’t show up in the previous years (so there is a recent spike in popularity)

 

CraftConf2016

Tim Steigert’s Closing Keynote at CraftConf 2016

See for Yourself

As a fun exercise in data visualization and trend analysis, I encourage you to try it out for yourself, using the CSV file produced by my script. To start with:

  • Load the dataset: Dataset > Web > CSV file (Keep “First line is header” checked)
  • Set the URL to the CSV file link above, and click Load
  • View as: Column
  • X Axis: Keyword
  • Sum by, Facet by: None
  • Color by: Keyword
  • Sort by: Frequency
  • Set the X axis bins to: 100

The typical way to drill down using SandDance would be:

  • Select a keyword (say “lean”)
  • Click Isolate. Everything else gets filtered out (note the “Filtered” count increased from 0 to 2)
  • Now you can check “Details”, or “Facet by…”, for example
  • To go back, simply click “Filtered” to clear the selection
CraftConfSanddanceArchAnalysis

Isolating the keyword “Architecture” in SandDance

Isolating the “Other” keyword will reveal a whole lot of keywords that don’t show up in the first 100 bins. You can also take the SandDance tour (by clicking on Tour) and discover many other interesting ways of playing with SandDance.

You can find the source code in my Git Repository WordFreqCount. If you find it useful, please feel free to reuse, derive from or improve it. Note that credit goes to Martin for the original code on web scraping using Python and Beautiful Soup, which I largely adapted from. And of course, thanks to Microsoft for making the elegant and powerful SandDance available for free!

Prezi’s 7 Year Itch?

This has been a really hard post for me to write, because some of the brilliant engineers and designers at Prezi are my professional acquaintances and/or good friends. And that’s the very reason I’m writing it – real friends tell it like it is. I’ve been putting this off since I first started using Prezi almost a year ago. But now that I have experienced it in free, Edu and Pro flavors, and created some pretty complex ones, the initial tiny niggles now seemed to have, let’s say, zoomed in (pun intended). So Prezi people: please consider this well-intended motivation.

Prezi was founded in 2009, and I imagine at that time it must have felt something like Wolfenstein 3D: technologically ground-breaking and in some circles, controversial (you may recall Microsoft tried to acquire id Software). Powerpoint needed to be killed, because it enforced linear thinking – and Prezi was the answer. Yet, 6 years later, linear thinking is still popular:

With 60 mn users (and 160 mn -/+ 18% Prezis created, depending on which part of their website you refer to), obviously they’ve got something right. Right enough that they are able to sustain an international presence on a single product. That rarely happens at this scale, and I suppose it is something to be admired.

The Platform

The in-browser experience is brilliant. Works well and virtually eliminates the entry barrier.  I tried using the desktop variant, Prezi for Mac, once. Just once. I found that it in terms of usability, it didn’t offer any major advantages over the website. But the real problem was that it kept false-flagging sync conflicts between the “local” and “cloud” versions. Eventually I decided the fake stress wasn’t worth it and switched back to the browser.

You may not have this dilemma if you would like to use the free power of Linux to share your ideas – Prezi doesn’t support it. (To be fair, they more than make up for this by actively promoting the tech and open source community in Budapest, by regularly organizing/hosting events, meetups and conferences).

There is an iPad version that “enables users to pan, and pinch to zoom in or out of their media” (That’s perfect, because that’s pretty much what an iPad is good for, anyway).

What I really wish for is the ability to resize the slide view pane on the left, so I can identify and jump to specific slides. And multi-select, especially for reordering (use the Force of the Shift Key). I would also like to mark some sections so I can focus on certain areas for preparation. Or even skip straight to one depending on how the conversation is flowing.

I mentioned stress earlier. You can quite easily recreate it by using 2-finger zoom gesture on a Mac. It’s not very precise and often ends up being a lot more than you’d expect. I once had to spend hours fixing my slides when a select-move-zoom operation didn’t go as planned. I learnt the hard way to stick to the zoom buttons and not use gestures.

And finally, it all runs on Flash. Yes, even now.

Features

The undo/redo feature also has a bit of a mind of its own. For simple operations it usually works fine, but when you’re in the flow and make a mistake, it often does the thing you’d least expect. Infinite undo/redo shouldn’t be that hard to implement either, considering everything is an object with attributes, anyway. What would be really nice, though, is some kind of checkpoints — major versions that you can roll back to, branch out of, or share with someone (e.g. numbered drafts). Because, 2015.

The guides when moving stuff around are hard to see. Too often they blend into the background, especially e.g. if the background has a grid. Get those ants marching again!

I also miss the ability to add notes, although what I would really expect is a more modern way of representing “prompts”, in a way that would be intuitive for presenters.

Warning: Brutal Honesty Ahead

Try this test, Prezi folks: create a new Prezi, single slide only, and export it to PDF.

  • Number of pages expected: 1
  • Number of pages found: 2.  The first page is always duplicated during export, regardless of number of slides.
  • Test result: FAIL.

Really, nobody noticed this in all these years?

Or that the image borders provided by the Aviary plugin are too thick and can’t be adjusted? (I trust you when you say my “private”, potentially proprietary/copyrighted images are truly private, even though there seems to be no obvious ToS or protection mechanism).

I’ve had Aviary and PDF export error out on me on a couple of occasions. I can live with that. But must you call them IO_ERRORs? That sounds more like the server ran out of disk space or something. And to non-programmers like my Dad, it probably looks like “ten errors” misspelt.

Nice to have: compressed PDF, with lowered quality. Try printing a 60 MB PDF some time, or re-uploading it — when just a link would have sufficed.

For that matter, why limit social sharing options to only to Facebook? How about Twitter, LinkedIn and dare I say, SlideShare integration as well? (PS: Do people really share presentations on Facebook? How about a good old “Email this” shortcut?)

There is tremendous potential to use Prezi as a content generation tool if it allowed exporting to video, or directly to YouTube (and/or Vimeo). Think tutorials, troubleshooting guides, walkthroughs and just plain not having to talk over and over again — present it, record it, share it. Other people seem to have the same idea, too.

And if only there was a way to present to the second screen, so in a presenter view (or just Edit mode) I can see what’s coming next and how I’m doing with my timing.

Use Your Imagination

sod2

Animated GIFs are still the quickest way of demonstrating processes in action, for example. Importing Google Spreadsheets and converting them into graphs within the editor. Powerful timelines like TimeGlider. Simple maps like OpenHeatMap. Leveraging the zooming and object hierarchy features to build a MindMap mode (which may even serve as a foundation for the final Prezi). And why limit the zooming and rotating to 2 dimensions only? I would love to make a Prezi with 6 slides, each represented on a face of a cube which can be rotated in 3D space. After all, the future of storytelling is going to be virtual and immersive.

(Here’s an idea: API and plugins).

Prezi is great, but that’s no reason to not be even greater.

(Or just release the Kraken: make it open source, let us fork it and watch how people use the platform to build even more clever ideas for telling their stories).

 

 

 

Amuse UX Conference, Budapest

Last week, I had the privilege of being part of a group attending the first edition of AmuseConf on behalf of our company. Amuse is “an international conference for anyone interested in how to design and develop successful products that users love”. It’s organized by the same good folks that bring us the outstanding CraftConf year after year, sponsored primarily by Prezi and UStream (and SAP in case of Amuse). They did a near-perfect job, with only minor glitches with the seating and catering on the first day. Considering that the Big Data oriented CrunchConf was also literally next door, the event was practically flawless. Fast, uninterrupted WiFi and no food options for vegetarians/vegans remained a hallmark this organizing team (even though Tom Illmensee, event MC is himself vegetarian 😉 ).

(BTW, if you’re wondering why so many tech conferences are being hosted in Budapest, the event’s WiFi password should give you a hint):

20151103_083708867_iOS

510 attendees from 32 countries (as far away as Australia) made Amuse a roaring success, as did its impressive lineup of Speakers27% of the speakers were women, which is great for a tech conference — I hope next year we have even more!

Below is a summary of the talks I found the most relevant to my work. But by no means does that mean you should skip the other talks… depending on where you are and what you’re doing, you might be interested in some of the eclectic topics covered such as:

  • Designing web interfaces for children by Trine Falbe
  • Conducting research outside “sample of convenience” by Bill Selman from Mozilla Foundation
  • Design Thinking by Tobias Haug of SAP (my favorite quote: “Innovation = Execution x Creativity”)
  • How to get your dream UX job by Andrew Doherty of Google (worth checking out just for his mad presentation skills)
  • The Ethical Designer by Cennydd Bowles
  • Storytelling in a multidevice landscape by Anna Dahlstrom

Design Equilibrium

By Jonathan Lupo

Jonathan opened the conference with a very engaging talk drawing parallels between businesses and ecosystems: a “balanced exchange of value between Actors, Enterprise and Brand”. He gave practical examples citing the application of Lynn Shostack’s work on Service Blueprinting to a transformation in the healthcare industry. I strongly encourage viewing his inspiring talk on YouTube.

His core suggestion is a separation of Product Design from Service Design. The latter “fills in whitespaces between points of [rich] engagement provided by products”, helping to restore balance to the overall experience, and hence the business ecosystem. This is the real intangible value of services, as opposed to products.

He also proposed the concept of an “Engagement Model”: a framework to contextualize all the data a business collects.

UX: Design as a Science

By Joel Marsh, author of the UX Crash Course

Joel’s key message was that “Scientific UX Design is reproducible”: essentially drawing on the principles of the Lean Startup and applying them to the UX domain. His presentation was one of the most popular and engaging ones, and his quotes and examples garnered a lot tons of positive feedback. One thing that struck me was his exposition on the two types of creativity: Creative expression and creative problem solving. He noted that an over-applicability of creative expression can make you feel good as a designer, but result in an over-designed and bloated product:

ArtVsDesign

Another talk I would highly recommend watching when it comes out on UStream.tv.

Making Dog Food a Part of Your Balanced Diet

By Toby Sterrett

Toby used his work at Simple Bank to highlight the pros and cons of “eating your own dogfood”. The initial employees used the app themselves, and one of the downsides was that the missed revelation that users of such a smooth app had to deal with a paper form-based process to close their account, which took up to 20 days.

Another inspiring talk that you should definitely check out, full of quotes of wisdom like:

  • “Delight is design’s superpower”
  • A past discussion on Leadership strategy: “Build a shared vision, get the **** out of the way”
  • “UX is not about throwing technology at a problem, but throwing people at a problem”

On the other hand, Simple A/B tested as many as 16 variations of their login page (for more examples, check out UserOnboard.com).

Live posters being created by @remarker_eu

Live posters being created by @remarker_eu

How We Built Hotjar and Onboarded 50k Users in a Year

By Dr. David Darmanin

David used practical examples from Hotjar to support his model of “Drivers, Barriers and Hooks” when dealing with site visitors. He also put a quirky twist on some timeless wisdom:

The two most amazing insights for me were:

  • Hotjar captures every single customer interaction on a Trello board, and uses that feedback to prioritize their features.
    • They also make their roadmap public, which demonstrates their commitment and at the same time reduces enquiries about feature requests
  • They use the income from their paid customers to fund the creative freedom to build features for their free customers

The Invisible Interface: Designing the Screenless Experience

By Avi Itzkovich

Avi, founder of UXSalon, opened with a discourse on recent editions of Microsoft Productivity Future Vision. From there he led the discussion on towards a future without bigger and wider screens (which wouldn’t require “superhuman arm strength”):

  • “The most profound technologies are the ones that become invisible” 
    • Like automatically opening sliding doors
  • “Voice UI is the future”
  • “Gesture control is here to stay, but not on screens”

The Best Interface is No Interface

By Golden Krishna

Golden surmised that we are all “chipping away at digital chores”, and we don’t have to be “slaves to screens”. He has laid the foundations of the #NoUI movement with his book“The Best Interface is No Interface”. His excellent talk (slides here) was supported by book reading and real examples. Also, don’t forget to check out his accompanying toolkit on “how to create elegant solutions with no screens”.

For further inspiration to join the movement,  take a look at his Producthunt collection of “interfaces that require little or no time with screens”.

Magical UX and the Internet of Things

By Josh Clark

Josh opened with an announcement of his book release: “Designing for Touch”. His presentation was literally magical, complete with a wand, to the point that he managed to tie in together excerpts from preceding talks and put the whole conference in perspective. I found a similar slidedeck from one of his previous talks here, and I highly recommend taking a look at it while we wait for the official conference videos to come up on Ustream.tv. It was a treasure trove of out-of-the box examples like:

  •  Augmented REality Sandtable (ARES), which literally turns dirt into a high-tech, military-grade user interface… using not much more than a Kinect and projector
  • Grab Magic, which brings superpowers to data transfer
  • Propeller Health, which connects Asthma inhalers to phones for health monitoring

Josh’s key message was “interaction at the point of inspiration”: that we should think of “the whole world is an interface, just like it has always been”. He proposed “thereables” instead of wearables: bits of smart technology in the physical space where we would expect to interact with them, not something we burden ourselves by carrying or wearing all day long. To this end, he suggested that “the smartphone is Magic Wand 1.0 for everyone” and we should start thinking of it as just more than a screen.

IMG_4185

Regarding user interfaces, he had 3 bits of advice that I found remarkable:

  • “Technology should amplify our humanity”
  • “We shouldn’t educate users on how technology works, unless we really *have* to”
  • “Honor intention, don’t assume it”

Josh ended with a call to action:

Like this one.

 

 

 

 

 

Takeaways from Craft Conference 2014, Budapest – Day 3

Continued from Day 2, here are the talks from Day 3 in order of my personal preference (and relevance) which may differ from yours:

Complex Projects aren’t planable but controllable

by Jutta Eckstein (Slides | Video)

Sadly, Jutta’s slides aren’t available online but the talk was packed with solid advice for Project Managers. Some of them were:

  • Our predictions are usually based on a coherent story, not a complex, dynamically changing reality
  • Targets should be ambitions, not absolute
  • Focus on the value gained, rather estimates of the effort required
  • Having an  annual budget is like having a bank that opens for business only once a year
  • Annual budgets are not optimal because they are never underutilized: the excess (if any) is still used up, never returned back
  • Consider [event-based] rolling budgets, rolling plans and rolling control
  • Check value and progress regularly
  • Don’t just trust the experts, seek feedback from diverse groups
  • Have different planning strategies for Roadmap (themes only), Release (features based on value & velocity) and Iteration (Stories based on value (+estimate) & velocity)

Recommended reading:

Architecture War Stories

by Stefan Tilkov (Slides | Video)

Probably the most amusing talk of the event. Stefan shared some hilarious real-life examples of architectural disasters… some of them still very much in use (of course no names were revealed). His advice was to go back to the basics:

  • Make data free of code dependencies
  • If it makes you want to pull your eyes out, maybe you shouldn’t be doing it
  • Better ask for forgiveness than permission
  • Development environment is not the same as production environment
  • Feedback, reflection, iteration

Responsibly maximizing craftsmanship in software engineering

by Theo Schlossnagle (Slides | Video)

Theo revisited the basic problems plaguing software development today and went on to share some of his experiences at Circonus. One of the tips was to treat all software you write as open-source within the organization, even if you don’t plan to release the source outside. Other highlights:

  • Turtles all the way down: Software at scale is tied together with loose string & hope
  • We use human language to describe software specifications, and this can be interpreted by different people in different ways (like poetry)
  • Technical debt is non-linear; large monolithic components are more likely to fail because they are hard to maintain
  • Reusability is good, but use the right tools for the right purpose. Accept that that right tool may not yet exist and may need to be written
  • Diversity is an emergent property of scale
  • Thou shalt be judged by your API

For software developers, the message was loud and clear:

kPhone_IMG_0951

Data-Driven Software Engineering

by Jevgeni Kabanov (Slides | Video)

Jevgeni went thorough the effort of analyzing 1000+ projects on Productivity, Quality and Predictability. Sadly the slides aren’t available, but the raw data they were based on is. You can still watch the video and grasp the most important points, such as the fact that the best software projects are delivered 80% of the time on schedule and still have 25% critical issues. Or that automated tests improve quality by 26%. There were also secondary interpretations such as code reviews improve architecture, not just code quality.

Jevgeni emphasized on the importance of measure & experiment over simply “doing agile”. Some suggestions for measurement were:

  • Deadline misses
  • Scope changes
  • Blockers after release
  • User satisfaction

The presentation became a bit controversial later but as Jevgeni said: “People, chill – I gathered data and presented my analysis – feel free to take the data and do a better job”. And I think this talk had the most creative closing slide ever 😉

Without Present or Past: How to Think Distributed – The Hard Way

by Endre Sandor Varga (Slides | Video)
By far the most profound and philosophical talks at Craft. If you have an interest in Distributed Systems, AI or Information Architecture, I would strongly recommend watching the video. The core concept was based on applying epsitemic thinking to systems that are distributed, concurrent, able to fail independently, and communicate over a lossy medium with non-deterministic communication delay. Endre touched upon:
  • Two Armies Problem
  • The final important message
  • Difference between actual state and observed state
  • The Omnipotent Observer (doesn’t exist, because state is always queried)
  • Global introspection and self-awareness
  • Cone of the past
  • The present is volatile and subjective
  • Going from someone knows -> someone knows everyone knows -> everyone knows everyone knows

Endre ended with this advice:

  • Have a properly defined failure model
  • Never assume reliable communication
  • Never assume common knowledge

Functional Reactive Programming in Elm and JS

by Evan Czaplicki (Slides | Video)

Evan is the designer of the Elm language and his enthusiasm is infectious. Over the course of his talk, he built and demonstrated a simplified (yet slick) Mario game complete with physics and reaction to keyboard inputs. Elm is a Functional Reactive Programming language for web browser based GUIs and this game was a fine demonstration of concepts that Bodil Stokke and Jonas Boner touched upon in their respective talks.

There is no doubt that Elm is a game-changer. Here is another example of traffic simulation.

Further reading:

Find the Right Abstraction Level for Your Tests

by Gerard Meszaros (Slides | Video)

If you’ve ever been haunted by the question of what level of testing is enough (and who hasn’t?) then this presentation was for you. The key message was to think in terms of not what you can add, but what you can leave out of the test: “if it isn’t essential to conveying the essence of the behavior, it is essential to not include it”. Over 39 slides, Gerard illustrated this step-by-step with one, continuous, easy-to-follow example.

Software Psychology: The Art of Listening to Code

by Bjorn Freeman-Benson (Slides | Video)

Bjorn talked about the concept of code screams: behavioral indications of a deeper problem in the system. For me the key takeaway was that you should continuously monitor your processes as well as your systems in production (e.g. by gathering usage statistics) and fix the root cause when a problem is found.

Others

Here are some talks that I missed, but which received a lot of positive feedback. Thanks to ustream.tv they are available online. Also check out #CraftConf on Eventifier.

Jackstones: the journey to mastery

by Dan North (Slides | Video)

McDonalds, Six Sigma, and Offshore Outsourcing: Unexpected Sources of Insight

by Chad Fowler (Slides | Video)

Testing the Hard Stuff and Staying Sane

by John Hughes (Slides | Video)

The Better Parts

by Douglas Crockford (Slides | Video)

Functional Examples from Category Theory

by Alissa Pajer (Slides | Video)

 

Takeaways from Craft Conference 2014, Budapest – Day 2

I had the privilege of attending the speaker sessions of Craft Conference last week, the central theme of which was software craftsmanship. There were many inspiring talks and so was the venue. Think of it as TED for software developers. The icing on the cake was free beer, complimentary Palinka, unlimited coffee and a blues band. 20% of the speakers were women (but only 4% of the attendees) and ~350 of the 900+ attendees were from abroad. The event was virtually flawless. The usual systems that breakdown at scale: WiFi, food and toilets, all just worked. Plus there were small thoughtful touches such as English translations of useful Hungarian phrases on the attendee badge. Everything about the conference was impressive, including the raffle prizes which included a R/C drone!

kPhone_IMG_0936

Other interesting features were the use of Sli.do to manage audience questions in real-time, and a live-projected twitter feed. I had the opportunity to interact with practicing or aspiring software crafts[wo]men from Ukraine, Japan (ok, technically SF), Netherlands, Sweden, North America, UK and… India!

I’ve tried to distil out the summary of talks that interested me. They are ordered by my preference, not the order in which talks were actually conducted. These are my interpretations and my views, so please bear in mind that they could be wrong or biased. There were 3 tracks (parallel talks) over 2 days, so essentially I attended only about 1/3rd of the total. Day 2 was the first day of the speaker talks… I didn’t attend day 1, which was workshops.

I strongly recommend checking out the agenda and viewing the talks that interest you online, you might find some that I didn’t attend but are of direct interest to you. To me the top 3 recurring themes of the conference seemed to be:

  1. State management in complex and distributed systems
  2. Better automated testing & TDD
  3. The abuse of agile (in development and in project management)

How I Learned to Stop Worrying and Love Flexible Scope

by Gojko Adzic (Slides | Video)

Probably one of the most-loved and honest talks at Craft. The storytelling was simply mind-blowing (an example of agile from 1628 AD, Ducati’s experience with the Second System Syndrome, threats to competitive advantage from Google & the Russians) and so was the message. I would highly recommend watching the video, Gojko is an outstanding speaker. The highlights were:

  • The most common software development methodology these days is WaterScrumFall, in which all the essential planning is done upfront by management and the development is done in a [predictable] number of sprints
  • This is because the concept of agile is not attractive to management, unless they truly believe in keeping scope flexible. Therefore agile generally remains underutilized.
  • Agile is not just about continuous delivery, but also continuously reacting to local, temporal and human factors
  • Project plans and roadmaps should not be linear, but literally a “map of roads”: multiple options with selection criteria (like a GPS). A roadmap with a pre-decided outcome is not a roadmap, but a road in a tunnel.
  • Try new things, at a survivable scale and select the ones that work (Throwing away bad code is a way to reduce your technical debt)
  • Don’t just ship software, make an impact
  • On the topic of outsourcing: Usually the objective is to minimize costs, so the focus is not on excellence or flexible scope
  • On User Stories:
    • Are a way to delegate details
    • Avoid translating the product roadmap into a set of JIRA stories, instead consider hierarchical backlogs
    • Add a victory condition to your user stories, which is related to changing user behavior rather than complying with the existing behavior (e.g. Monitor inventory *faster*)
    • A good user story is a survivable experiment

Recommended reading:

Agility and the essence of software architecture

by Simon Brown (Slides | Video)

This was also one of my favorite talks. I like to think of it as Minimum Viable Architecture for developers. Simon is an inspiring speaker, more so because he eats his own dog food. The premise was simple:

  • Agile delivery does not imply agile architecture
  • Agile development still requires upfront thinking to define the overall architecture
  • The team must have a shared, consistent vision of the significant structural elements of the product
    • With this pragmatic tradeoffs can be achieved: e.g. a monolithic deployment container, containing microservices
  • UML was supposed to solve this problem, but it is poorly understood, not widely adopted and has its own overheads

The solution: NoUML! Abstraction is more important than notations. A team can define their own legend for these abstractions. Design diagrams are supposed to reduce complexity (through abstraction), not increase it. The 3 things that the team needs to have a common understanding of are:

  • Structure
  • Risks
  • Vision

In real life, we rarely have 1:1 mapping between design diagrams and code. In general, a hierarchical C4 architecture diagram can solve this problem:

  • Context
  • Container
  • Components
  • And optionally, Classes

And yes, while good architecture is a shared responsibility, it is important that only one person (or a small group of people) are responsible for maintaining the overall architecture definition.

Bring Software Architecture Back!

Recommended reading:

Getting Things Done at Scale

by Amber Case (Slides | Video)

This was a talk I could relate to a lot, because of GTD and the differences in corporate cultures of large, old organizations vs smaller, newer ones. Amber is a TED speaker, and it clearly shows. She shared her experiences during (and after) the acquisition of Geoloqi by ESRI.

Only 1 of 50 people she spoke to for advice had a happy acquisition experience, and the main reasons were:

Crippling management/overhead to get simple things done Culture clash
Founder flight Jealousy/blocking from parent company employees
Lack of detailed transition plan Sprinters vs Marathon Runners
Loss of passion for original product Loss of respect and cross-compromise

The solutions discussed were:

  • Pre-negotiate, don’t be vague. Predetermine your outcome.
  • Learn the local language (technical terms, tools, company culture)
  • Win friends to influence people. Develop trust.
  • “Beta test” people via small projects. Best code wins.
  • Scale teams down from unmanageable numbers to 5-6 “doers”
  • Communicate. Respect. Give first.
  • Small revolutions
  • Distribute stress

An interesting concept was the creative use of IRC bots, e.g. to send out a daily email summary of accomplishments (!done). Towards the end, she also shared an effective way of “organic hiring”: turning contractors into full-time employees.

It’s never too late to fight your legacy!

by Mate Nadansi (Slides | Video)

Mate delivered a very strong and reassuring message: Legacy code isn’t bad, just old or over-iterated… and, with some sensible planning, foresight, and many iterations of hard work, it can be replaced by more modern code. He explained how they achieved exactly that at ustream.tv. The presentation requires some familiarity with web programming but it would still benefit anyone working with legacy code… because maintaining legacy code builds character. 😎

Programming, Only Better

by Bodil Stokke (Slides | Video)

The core message was about how the introduction of multiple [mutable] states makes programs unmanageably complex. Topics included:

  • Referential Transparency
  • Representation of state using numbers instead of objects to it remains immutable
  • “Encapsulated state” is still state
  • Effect on concurrency
  • Additional complexity added by control structures in contemporary program

The presentation itself was a work of art and for me the highlight was that Bodil was editing & running code from within the slides!

Recommended reading:

  • Out of the Tar Pit, a 2006 paper about Functional Relational Programming by Ben Moseley and Peter Marks

Going Reactive: Event-Driven, Scalable, Resilient & Responsive Systems

by Jonas Boner (Slides | Video)

Jonas gave an inspiring and pragmatic talk about how the nature of, and expectations from, applications have changed dramatically over the years. The highlight was that not only did he distill them into the 4 attributes mentioned in the topic of the talk, but also shared valuable insights into how to practically achieve them from a technical standpoint. Some of the approaches discussed were:

  • Loose coupling
  • Never blocking
  • Asynchronous operations
  • Actors and Agents
  • Futures in Scala
  • Minimizing contention
  • Maximizing locality of reference
  • Single Writer Principle
  • The relation between scaling up and scaling out
  • Decoupling error handling from business logic
  • Bulkheading to prevent cascading failures
  • Maintaining consistent latency across blue skies conditions, high traffic and failure

Recommended reading:

What Makes a Good Development Process?

by Bruce Eckel (Slides | Video)

Bruce Eckel is an industry veteran an author who needs no introduction (I read his textbooks in high school). I highly recommend going over his slides, they are very comprehensive, thoughtfully put together and highly informative. He has already distilled years of experience into a few pages, and I don’t think I can add any more value except quoting the one thing that stood out for me: The cognitive load of carrying tensions prevents us from doing creative tasks well.

Acknowledging CAP at the Root – in the Domain Model

by Eric Evans (Slides | Video)

This talk dealt with a very specific type of problem (CAP = Consistency, Availability & Partition tolerance), and solving it using a domain-driven programming model. One of the interesting concepts was that of defining aggregates within a distributed system. Instead of trying to keep the entire system in a consistent state, the contents of each aggregate are guaranteed to be consistent (even if equally stale). There is only a single point of reference into an aggregate. Aggregates are contained within a bounded context and transactional consistency must not cross these boundaries. Domain events interpreted within this context cause state changes. Eventually overall system consistency can be achieved through synchronization according to a reasonable SLA.

Don’t forget to check out Day 3!

FilesDesk

2012

This started out as a side project between a couple of us at work to sharpen our C#/XML skills and generally do something challenging. A C#.NET Application + Service that enables browsing files on disk by Tag (as opposed to Path). Typical usage:

  • A Tag Cloud for your hard disk
  • Add/remove tag(s) to selected file(s)
  • List & Filter files by tag(s)
  • Monitor specified folders (ex: My Documents) for new/deleted/renamed files
  • Portable XML Database

Useful for Researchers, Writers and Obsessive-Compulsive File Hoarders.

DOPE: Distributed OPerating Environment

November 2003

I did this project with a friend around that time that Grid Computing was getting popular in the mainstream with SETI@Home (in its pre-BOINC avatar).

A Client-Server Java app that used Distributed Computing principles to:
a) Break down a given, computationally intensive task into chunks*
b) Distribute these chunks to clients on the network, favouring those with lower latency
c) Re-assemble the results returned by clients into a single unit
d) Re-send chunks for which result was not received within stipulated time

*Limitations apply: this was not a general-purpose implementation