An Exercise in Analytics: Using Microsoft SandDance to Visualize Trends at CraftConf

I’ve been lucky to attend every CraftConf Budapest since its inception in 2014.  It has always been a mind-expanding experience, with a healthy mix of established and newcomer speakers from the tech industry worldwide. Although Craft, as the name suggests, is focused on Software Craftsmanship, the specific topics of talks vary each year as industry trends fluctuate. I was interested in taking a deeper look at trends of technologies and paradigms as they become more or less popular over time, and to find those, if any, whose popularity has remained more or less constant.

While doing some research into data analytics for a project at work, I came across Microsoft SandDance. My interest in the CraftConf data was the perfect opportunity to teach myself SandDance (and some Python as well). So I put together a quick experiment: Using Martin’s excellent tutorial on Web Scraping with Python as a reference, I wrote  a Python script that scrapes CraftConf’s talk descriptions (using archives from 2014, 2015 and 2016) and produces a CSV file of the most frequently occurring words. Obviously standard English language words like pronouns, days of the week, and so on are ignored. What we’re left with is a list of top 100 words for each year, and their frequencies of occurrence, which can be visualized  in SandDance.

Although this approach is quick and more or less effective, the limitation is that it may not accurately reflect trends for phrases like “Agile Methodology” — the word frequency of “agile” may not be the same as that of “methodology”. But that’s something that can be worked on later. So although I would take this as a good indicator (which meets my purpose), I wouldn’t use the analysis outcomes as a serious reference.

What the Data Tells Us

Here are some interesting findings from the first pass (150 of the most frequent words selected, out of those 100 produced after filtering out common and punctuated words):

• “Product” showed up 26 times in 2015 and nearly doubled to 51 in 2016 (a growing trend: no talk, some talk, twice the talk…)
• “Functional” [programming] appeared 16 times in 2014, and not in the other 2 years (something that’s coming and going?)
• Similarly, “Architecture” showed up respectively 15, 37 and 23 times (up and down)
• “DevOps” was an equally hot trend in 2014 and 2015 but didn’t show up in 2016 (presumably because the hype is over)
• “Microservices” appears 29 times in 2016, but didn’t show up in the previous years (so there is a recent spike in popularity)

 

Tim Steigert’s Closing Keynote at CraftConf 2016

See for Yourself

As a fun exercise in data visualization and trend analysis, I encourage you to try it out for yourself, using the CSV file produced by my script. To start with:

• Load the dataset: Dataset > Web > CSV file (Keep “First line is header” checked)
• Set the URL to the CSV file link above, and click Load
• View as: Column
• X Axis: Keyword
• Sum by, Facet by: None
• Color by: Keyword
• Sort by: Frequency
• Set the X axis bins to: 100

The typical way to drill down using SandDance would be:

• Select a keyword (say “lean”)
• Click Isolate. Everything else gets filtered out (note the “Filtered” count increased from 0 to 2)
• Now you can check “Details”, or “Facet by…”, for example
• To go back, simply click “Filtered” to clear the selection

Isolating the keyword “Architecture” in SandDance

Isolating the “Other” keyword will reveal a whole lot of keywords that don’t show up in the first 100 bins. You can also take the SandDance tour (by clicking on Tour) and discover many other interesting ways of playing with SandDance.

You can find the source code in my Git Repository WordFreqCount. If you find it useful, please feel free to reuse, derive from or improve it. Note that credit goes to Martin for the original code on web scraping using Python and Beautiful Soup, which I largely adapted from. And of course, thanks to Microsoft for making the elegant and powerful SandDance available for free!

CES 2016: From the Fringes

The Consumer Electronics Show in Las Vegas is the premier trade show for tech product previews and release announcements, going as far back as the VCR in 1970 to Driverless Cars in 2013*. This year the CES featured about 3800 exhibitors, spanning 2.47 million sq. ft. spread out over 3 locations** visited by 170,000 media and industry professionals — and I was privileged to count myself amongst them. Featuring keynote addresses from Intel, Netflix, IBM, Samsung, nVidia, Volkswagen and other big names, a lot has been written, presented and shared on mainstream as well as social media about the 4 day event. This chart sums it up the hype pretty well:

Source: BuzzRadar, CTA

I decided to share some of my views from the fringes, rather than the trenches — there is no point in rinsing and repeating what is already out there, nor do I have any delusions about the value of my personal opinion about tech that enables your car to count how many oranges are left in your fridge (yes, it was demoed, with voice control).

The Oculus Rift demo was by far the hardest to get into — there was a line, a line to get in the line, and a third holding area. Eventually I made it on the last day, and it took me about 20 min to recover from the simulator sickness caused by piloting EVE Valkyrie’s spacecraft from a living room chair. I still felt there were rough edges and the HTC Vive was by far a more refined, immersive and truly flawless experience. The new Sony PlayStation 4 VR was quite impressive as well: I could lean out of a moving car and look behind me, and the granularity of control was so good I could rotate knobs on the car stereo. OSVR.org based devices were quite popular too, and some others that caught my eye were Virtuix Omni active VR platform, AntVR’s Holodeck concept and ICAROS‘ EUR 10,000 gym equipment that lets you fly around in a virtual world powered by your own body. Certainly beats playing first person shooters wearing VR googles on a treadmill, or riding a virtual horse on a exercycle.

There were tons of clones (mostly based on Gear VR) and drones. Augmented Reality seems to be gaining ground, but despite solutions like the Sony SmartEyeglass and Daqri Smart Helmet, VR seems to be more popular of the two. It’s worth noting that virtually every VR or AR demo was running on Unity3D content, including those at NASA and IEEE’s booths.

I also tried my hand at racing simulators of various scales: from small VR setups, to actual cars mounted on motion platforms, to a massive 4×4 grid of 55″ OLEDs in front of a force feedback seat rig. There were several interesting display technologies on show: 3D without glasses, transparent (scaling up to entire walls), curved and Samsung’s modular, edge-blending display tech straight out of a sci-fi movie. Avegant’s Glyph might turn the display industry on it’s head, though, much like the way it’s worn.

 

On the automotive side, voice, gesture and intent based user interfaces seem to be gaining ground. Also making an appearance were adaptive user interfaces and improvements in sensor fusion, self-learning and self-driving techniques. There were tons of wearables, 3D printing and home automation booths. The two core themes seemed to be a maturing of the ecosystem (just about everything built on top of something else, not too many technologies solving problems from scratch) and apps for doing things that don’t need apps, like locking your front door. You’d think we would stop there, but no:

On the social innovation side, I found GrandPad, Casio’s 2.5D printing and the Genworth R70i Aging Experience very thoughtful. Besides these, I liked Mixfader‘s idea of an MVP slider for mobile DJs: after all, the crossfader is the main thing that requires precise tactile control, everything else can be relegated to the screen. Also impressive was Sony’s line of 409,600 ISO see-completely-in-the-dark cameras. And this is now a thing:

You’d also probably be able to find a lot of beautiful photos of Las Vegas on the Internet, so let me leave you with this video of a not-so-common Las Vegas activity that I squeezed in on the last day, courtesy of DreamRacing.com (very fringe-y because I picked a Nissan over a Ferrari). Thanks for reading!

* Apple, Google and Microsoft have their own tech events and despite the Xbox (2001) and Android devices (2010) being unveiled at CES, these companies tend to keep their product announcements exclusive to their own events. So no Hololens at CES.

** „Tech East (Las Vegas Convention Center), Tech West (Sands/Expo at the Venetian,  The Palazzo, Wynn and Encore) and Tech South (Aria and Vdara)

Amuse UX Conference, Budapest

Last week, I had the privilege of being part of a group attending the first edition of AmuseConf on behalf of our company. Amuse is “an international conference for anyone interested in how to design and develop successful products that users love”. It’s organized by the same good folks that bring us the outstanding CraftConf year after year, sponsored primarily by Prezi and UStream (and SAP in case of Amuse). They did a near-perfect job, with only minor glitches with the seating and catering on the first day. Considering that the Big Data oriented CrunchConf was also literally next door, the event was practically flawless. Fast, uninterrupted WiFi and no food options for vegetarians/vegans remained a hallmark this organizing team (even though Tom Illmensee, event MC is himself vegetarian 😉 ).

(BTW, if you’re wondering why so many tech conferences are being hosted in Budapest, the event’s WiFi password should give you a hint):

510 attendees from 32 countries (as far away as Australia) made Amuse a roaring success, as did its impressive lineup of Speakers. 27% of the speakers were women, which is great for a tech conference — I hope next year we have even more!

Below is a summary of the talks I found the most relevant to my work. But by no means does that mean you should skip the other talks… depending on where you are and what you’re doing, you might be interested in some of the eclectic topics covered such as:

• Designing web interfaces for children by Trine Falbe
• Conducting research outside “sample of convenience” by Bill Selman from Mozilla Foundation
• Design Thinking by Tobias Haug of SAP (my favorite quote: “Innovation = Execution x Creativity”)
• How to get your dream UX job by Andrew Doherty of Google (worth checking out just for his mad presentation skills)
• The Ethical Designer by Cennydd Bowles
• Storytelling in a multidevice landscape by Anna Dahlstrom

Design Equilibrium

By Jonathan Lupo

Jonathan opened the conference with a very engaging talk drawing parallels between businesses and ecosystems: a “balanced exchange of value between Actors, Enterprise and Brand”. He gave practical examples citing the application of Lynn Shostack’s work on Service Blueprinting to a transformation in the healthcare industry. I strongly encourage viewing his inspiring talk on YouTube.

His core suggestion is a separation of Product Design from Service Design. The latter “fills in whitespaces between points of [rich] engagement provided by products”, helping to restore balance to the overall experience, and hence the business ecosystem. This is the real intangible value of services, as opposed to products.

He also proposed the concept of an “Engagement Model”: a framework to contextualize all the data a business collects.

UX: Design as a Science

By Joel Marsh, author of the UX Crash Course

Joel’s key message was that “Scientific UX Design is reproducible”: essentially drawing on the principles of the Lean Startup and applying them to the UX domain. His presentation was one of the most popular and engaging ones, and his quotes and examples garnered a lot tons of positive feedback. One thing that struck me was his exposition on the two types of creativity: Creative expression and creative problem solving. He noted that an over-applicability of creative expression can make you feel good as a designer, but result in an over-designed and bloated product:

Another talk I would highly recommend watching when it comes out on UStream.tv.

Making Dog Food a Part of Your Balanced Diet

By Toby Sterrett

Toby used his work at Simple Bank to highlight the pros and cons of “eating your own dogfood”. The initial employees used the app themselves, and one of the downsides was that the missed revelation that users of such a smooth app had to deal with a paper form-based process to close their account, which took up to 20 days.

Another inspiring talk that you should definitely check out, full of quotes of wisdom like:

• “Delight is design’s superpower”
• A past discussion on Leadership strategy: “Build a shared vision, get the **** out of the way”
• “UX is not about throwing technology at a problem, but throwing people at a problem”

On the other hand, Simple A/B tested as many as 16 variations of their login page (for more examples, check out UserOnboard.com).

Live posters being created by @remarker_eu

How We Built Hotjar and Onboarded 50k Users in a Year

By Dr. David Darmanin

David used practical examples from Hotjar to support his model of “Drivers, Barriers and Hooks” when dealing with site visitors. He also put a quirky twist on some timeless wisdom:

MVP is the smallest possible product you need for your customers to tell you that it sucks #ux #lean #amuseconf

— Kartik Sachdev (@survivalcrziest) October 29, 2015

Hire people smarter than you. It's intimidating but you'll learn something & in the end they'll make you look good #ux #amuseconf

— Kartik Sachdev (@survivalcrziest) October 29, 2015

The two most amazing insights for me were:

• Hotjar captures every single customer interaction on a Trello board, and uses that feedback to prioritize their features.
  • They also make their roadmap public, which demonstrates their commitment and at the same time reduces enquiries about feature requests
• They use the income from their paid customers to fund the creative freedom to build features for their free customers

The Invisible Interface: Designing the Screenless Experience

By Avi Itzkovich

Avi, founder of UXSalon, opened with a discourse on recent editions of Microsoft Productivity Future Vision. From there he led the discussion on towards a future without bigger and wider screens (which wouldn’t require “superhuman arm strength”):

• “The most profound technologies are the ones that become invisible” 
  • Like automatically opening sliding doors
• “Voice UI is the future”
• “Gesture control is here to stay, but not on screens”

The Best Interface is No Interface

By Golden Krishna

Golden surmised that we are all “chipping away at digital chores”, and we don’t have to be “slaves to screens”. He has laid the foundations of the #NoUI movement with his book: “The Best Interface is No Interface”. His excellent talk (slides here) was supported by book reading and real examples. Also, don’t forget to check out his accompanying toolkit on “how to create elegant solutions with no screens”.

For further inspiration to join the movement,  take a look at his Producthunt collection of “interfaces that require little or no time with screens”.

Magical UX and the Internet of Things

By Josh Clark

Josh opened with an announcement of his book release: “Designing for Touch”. His presentation was literally magical, complete with a wand, to the point that he managed to tie in together excerpts from preceding talks and put the whole conference in perspective. I found a similar slidedeck from one of his previous talks here, and I highly recommend taking a look at it while we wait for the official conference videos to come up on Ustream.tv. It was a treasure trove of out-of-the box examples like:

•  Augmented REality Sandtable (ARES), which literally turns dirt into a high-tech, military-grade user interface… using not much more than a Kinect and projector
• Grab Magic, which brings superpowers to data transfer
• Propeller Health, which connects Asthma inhalers to phones for health monitoring

Josh’s key message was “interaction at the point of inspiration”: that we should think of “the whole world is an interface, just like it has always been”. He proposed “thereables” instead of wearables: bits of smart technology in the physical space where we would expect to interact with them, not something we burden ourselves by carrying or wearing all day long. To this end, he suggested that “the smartphone is Magic Wand 1.0 for everyone” and we should start thinking of it as just more than a screen.

Regarding user interfaces, he had 3 bits of advice that I found remarkable:

• “Technology should amplify our humanity”
• “We shouldn’t educate users on how technology works, unless we really *have* to”
• “Honor intention, don’t assume it”

Josh ended with a call to action:

We have bigger problems to solve. @bigmediumjosh #amuseconf #ux #innovation

— Kartik Sachdev (@survivalcrziest) October 30, 2015

Like this one.

 

 

 

 

 

Craft Conf 2015, Day 3

Continued from Day 2, here is a summary of talks I attended on Day 3:

From the Monolith to Microservices: Lessons from Google and eBay

By Randy Shoup | Video | Slides

Another eye-opening presentation with valuable insights, such as the fact that [a big organization like Google] doesn’t need architects, it just needs standardized communication and standardized interfaces. And that one of the biggest mistakes people make with microservices is reflecting the provider’s model instead of the consumer’s model. I highly recommend his talk, because it is based on the analysis of several Silicon Valley giants, successful either in the past or the present.

Interaction Driven Design

By Sandro Mancuso | Video | Slides

Sandro’s presentation was full of real examples rather than just theory. I had never heard of the Walking Skeleton before. It was an interesting intersection of DDD (Domain Driven Design), MVC-type architectures and SOLID principles, leading up to a pragmatic way of structuring and packaging software projects. Other advice from Sandro included modeling behavior, not state and not necessarily representing repositories as first-class citizens.

WebSocket for the Real-Time Web and the Internet of Things

By Peter Moskovits | Video | Slides

Not only was it an amazing presentation with live demos, Peter was also fully prepared with a backup plan for everything – including a PDF version of his presentation. After a historical perspective & technical explanation of how WebSockets work, he jumped into Kaazing demos which you can also experience online here. The most interesting was a kind of MVP for disseminating airline telemetry data (here).

Why Is An API Like a Puppy?

By Ade Oshineye | Video | Slides

RESTful APIs are not the solution to all of the world’s problems: Ade was short, succinct and insightful. The title of his talk reflected the fact that an API is an expensive long term commitment, it’s not just about the initial cost of software development. He got a lot of attention when he revealed that Google’s most successful API is AdWords, and it’s SOAP, not REST. Although REST is theoretically good, it doesn’t usually fit well with the real world consumer’s way of thinking. Another one of his gems was that if your [public] API is not being spammed/abused, then either no one is using it, or it’s happening and you’re not aware of it.

Implementing the Saga Pattern

By Caitie McCaffrey | Video | Slides

There wasn’t anything interesting to me during this time slot, so I decided to go with this one just for the Halo reference. There was just one picture of Halo. And a lot of “so” and “like”.

Techniques and Tools For a Coherent Discussion About Performance in Complex Architectures

By Theo Schlossnagle | Video | Slides

Theo decided it would be a good idea to plaster all his slides with huge pictures of steak. Anyway, after establishing that User Experience is measured in milliseconds, and that performance is also about the time spent between service layers, he covered distributed tracing systems such as Dapper and Zipkin.

Great Engineering, Failed Product

By Marty Cagan | Video | Slides

Marty drew on decades of experience in Silicon Valley to summarize why great products and companies fail over and over again. I highly recommend watching his inspiring and insightful talk. Some of the things he touched upon while comparing successful and poorly performing teams:

• Customers and company executives are a bad source of product ideas, because they don’t know what’s technically achievable
• Developers are a good source, and so is Data (analytics, metrics, usage)
• Multi-billion dollar projects are not based on a Business Case accurately predicting future revenue
• Roadmaps are not a good indicator because Customers have other options available to them
• Think Time to Money, not Time to Market – which means more than one iteration is involved
• Product Managers are not mere [user] story writers – they need to have a deep understanding of the business, industry, customers and constraints
• Most teams work in a way that gives them probably 20% of the benefit of Agile Methodologies
• Value outcomes over output; think in terms of results, not projects
• Successful teams run as many as 20 MVP experiments in a week – even if it involves hardware
• Successful companies use an OKR approach to measure progress

• The four product development questions:
  1. Will the customers choose it? (Customer Validation)
  2. Will they be able to use it? (User Experience)
  3. Can we build it? (Feasibility)
  4. Can our stakeholders support it? (e.g. Legality)

______________________________________________

Craft Conf 2015, Day 2

I had the privilege of attending the second year of CRAFT, a tech conference in Budapest focused on software craftsmanship. The last year (which was the first time it was held) had completely blown my mind. A year later I still keep referring back to the talks and haven’t finished fully absorbing them and putting all those inspiring ideas into practice.

In short…

Craft Conf 2014 was better. The speakers came from a more diverse background, the talks spanned a multitude of unrelated topics and I remember it being very, very hard to choose from talks happening in parallel. Each minute spent there was a revelation.

This year, though, many of the talks seemed to be plug for a company or a product, in disguise. Certainly there were brilliant takeaways, but not at the same scale as the previous year.

In my opinion, 2014 was also held in a better venue, although the 2015 venue was outstanding too, as far as tech conference venues go. But the rooms were too far spread out (the map was inaccurate), the acoustics were bad everywhere except the Main Room and unlike 2014, the WiFi was not flawless. Lastly, there were far fewer food choices, longer queues, no bottled water (even for the speakers) and therefore a lot of glasses clanking.

On the positive side, the schedule was followed down to the minute, the live video streaming was smooth and considering the scale of the event (1300 attendees), everything was beautifully organized. I’m not complaining – it’s just that the first CRAFT had set a pretty high standard.

(Video and Slides links will be updated by next weekend, when they become available)

Agile Engineering in a Safety-Critical World

By Nancy Van Schooenderwoert | Video | Slides

“Instead of freezing the ocean, learn to ride the waves” – Nancy’s talk was mainly about how our need for predictability for effective coordination is at odds with our need for fast learning to handle unknowns. She pointed out that in the agile context, “Architecture is any design decision that you cannot easily change”.

There was the customary reference to WikiSpeed to dispel the myth that hardware changes can’t fit within 2-4 week iterations. And an interesting one to a paper called TIR45 from AAMI: Guidance on the use of AGILE practices in the development of medical device software.

Coding Culture

By Sven Peters | Video | Slides

Sven’s talk was both informative and inspiring. Some of the key takeaways:

• Innovation needs time
• Stop and celebrate wins, however small they may be
• Balance your passion for code with your passion for customers
• Turn your passion into product
• Value trust, autonomy and transparency (Atlassian achieves this by using chat over other communication means)
• Products come and go, culture stays

Take a look at Atlassian’s Mood App and Stash Reviewer Suggester.

Building Reliable Distributed Data Systems

By Jeremy Edberg | Video | Slides

This one was good, until we went deep diving into the NetFlix Simian Army, which was also good but could have been summarized in just one slide. One thing that stood out from Jeremy’s advice was to “build for three”, because if you can overcome problems there then the solution can be [more] easily scaled up to n.

Don’t forget to check out NetFlix Open Source Software Center.

Oh! You Pretty Tools

By Andrew Bayer | Video | Slides

Andrew gave an interesting talk about the role of internal tools and their developers in the organization, covering both the pros and the cons. For example while making the build or buy decision, consider the fact that people are more expensive than software. And some thoughtful insights, like how Integration Tests can double-up as the roadmap to your tool’s usage. He also revealed that Cloudera runs ~2000 Jenkins CI builds every day(!)

The rest of it was basically about, and lessons learned from, CloudCat.

Testing and Integration (The Remix)

By Ines Sombra | Video | Slides

Ines entertainingly summarized everything we know so far, and topped it off with new insights for a good measure.  She emphasized:

• The importance of lightweight short-lived branches so that CI is not overlooked
• The more likely a test is to fail, the sooner you should run it
• The testing of provisioning systems, such as Chef Recipes, too
• How test setup time and parallelization are the key factors in minimizing the testing cycle time

She recommended this talk about the Google Build System.

Her punchline was that CI is a predictor of professional maturity at the organizational and individual level, and she ended with a “rantifesto” about building a culture of quality.

Beyond Features: Rethinking Agile Planning and Tracking

By Dan North | Video | Slides

From Cutting to Curing: Dan presented the powerful and inspiring idea that maybe software engineering is more like surgery than the civil engineering principles that we currently use to manage it. Agile methodologies essentially optimize for predictability, and this not necessarily a good thing. He mused on how a 2-week sprint is just enough time for a mini-waterfall, and thus we are all basically whitewater rafting.

After reviewing where the Agile Manifesto has brought us, he set an ambitious new goal to sustainably minimize the lead time to business impact. 

He ended with:

• The role of Features, Delivery and Kaizen
• Schedule, Measure, Track, Showcase
• How Value Stream Mapping can reveal surprises like typically a piece of work spends upto 90% of it’s time waiting for dependencies

How To Save Innovation From Itself

By Alf Rehn | Video | Slides

For me, Alf’s talk was the highlight of the event. It was so good and so inspiring that I won’t even summarize it here. Go watch it!

______________________________________________

The day ended with a party thrown by EPAM, which included free beer, a DJ-saxophone duo and a surprise flashmob.

You may also want to read my review of Day 3 of Craft Conf 2015.

 

 

 

 

 

Takeaways from Craft Conference 2014, Budapest – Day 3

Continued from Day 2, here are the talks from Day 3 in order of my personal preference (and relevance) which may differ from yours:

Complex Projects aren’t planable but controllable

by Jutta Eckstein (Slides | Video)

Sadly, Jutta’s slides aren’t available online but the talk was packed with solid advice for Project Managers. Some of them were:

• Our predictions are usually based on a coherent story, not a complex, dynamically changing reality
• Targets should be ambitions, not absolute
• Focus on the value gained, rather estimates of the effort required
• Having an  annual budget is like having a bank that opens for business only once a year
• Annual budgets are not optimal because they are never underutilized: the excess (if any) is still used up, never returned back
• Consider [event-based] rolling budgets, rolling plans and rolling control
• Check value and progress regularly
• Don’t just trust the experts, seek feedback from diverse groups
• Have different planning strategies for Roadmap (themes only), Release (features based on value & velocity) and Iteration (Stories based on value (+estimate) & velocity)

Recommended reading:

• Implementing Beyond Budgeting: Unlocking the Performance Potential, by Bjarte Bogsnes

Architecture War Stories

by Stefan Tilkov (Slides | Video)

Probably the most amusing talk of the event. Stefan shared some hilarious real-life examples of architectural disasters… some of them still very much in use (of course no names were revealed). His advice was to go back to the basics:

• Make data free of code dependencies
• If it makes you want to pull your eyes out, maybe you shouldn’t be doing it
• Better ask for forgiveness than permission
• Development environment is not the same as production environment
• Feedback, reflection, iteration

Responsibly maximizing craftsmanship in software engineering

by Theo Schlossnagle (Slides | Video)

Theo revisited the basic problems plaguing software development today and went on to share some of his experiences at Circonus. One of the tips was to treat all software you write as open-source within the organization, even if you don’t plan to release the source outside. Other highlights:

• Turtles all the way down: Software at scale is tied together with loose string & hope
• We use human language to describe software specifications, and this can be interpreted by different people in different ways (like poetry)
• Technical debt is non-linear; large monolithic components are more likely to fail because they are hard to maintain
• Reusability is good, but use the right tools for the right purpose. Accept that that right tool may not yet exist and may need to be written
• Diversity is an emergent property of scale
• Thou shalt be judged by your API

For software developers, the message was loud and clear:

Data-Driven Software Engineering

by Jevgeni Kabanov (Slides | Video)

Jevgeni went thorough the effort of analyzing 1000+ projects on Productivity, Quality and Predictability. Sadly the slides aren’t available, but the raw data they were based on is. You can still watch the video and grasp the most important points, such as the fact that the best software projects are delivered 80% of the time on schedule and still have 25% critical issues. Or that automated tests improve quality by 26%. There were also secondary interpretations such as code reviews improve architecture, not just code quality.

Jevgeni emphasized on the importance of measure & experiment over simply “doing agile”. Some suggestions for measurement were:

• Deadline misses
• Scope changes
• Blockers after release
• User satisfaction

The presentation became a bit controversial later but as Jevgeni said: “People, chill – I gathered data and presented my analysis – feel free to take the data and do a better job”. And I think this talk had the most creative closing slide ever 😉

Without Present or Past: How to Think Distributed – The Hard Way

by Endre Sandor Varga (Slides | Video)

By far the most profound and philosophical talks at Craft. If you have an interest in Distributed Systems, AI or Information Architecture, I would strongly recommend watching the video. The core concept was based on applying epsitemic thinking to systems that are distributed, concurrent, able to fail independently, and communicate over a lossy medium with non-deterministic communication delay. Endre touched upon:

• Two Armies Problem
• The final important message
• Difference between actual state and observed state
• The Omnipotent Observer (doesn’t exist, because state is always queried)
• Global introspection and self-awareness
• Cone of the past
• The present is volatile and subjective
• Going from someone knows -> someone knows everyone knows -> everyone knows everyone knows

Endre ended with this advice:

• Have a properly defined failure model
• Never assume reliable communication
• Never assume common knowledge

Functional Reactive Programming in Elm and JS

by Evan Czaplicki (Slides | Video)

Evan is the designer of the Elm language and his enthusiasm is infectious. Over the course of his talk, he built and demonstrated a simplified (yet slick) Mario game complete with physics and reaction to keyboard inputs. Elm is a Functional Reactive Programming language for web browser based GUIs and this game was a fine demonstration of concepts that Bodil Stokke and Jonas Boner touched upon in their respective talks.

There is no doubt that Elm is a game-changer. Here is another example of traffic simulation.

Further reading:

• Go try Elm!

Find the Right Abstraction Level for Your Tests

by Gerard Meszaros (Slides | Video)

If you’ve ever been haunted by the question of what level of testing is enough (and who hasn’t?) then this presentation was for you. The key message was to think in terms of not what you can add, but what you can leave out of the test: “if it isn’t essential to conveying the essence of the behavior, it is essential to not include it”. Over 39 slides, Gerard illustrated this step-by-step with one, continuous, easy-to-follow example.

Software Psychology: The Art of Listening to Code

by Bjorn Freeman-Benson (Slides | Video)

Bjorn talked about the concept of code screams: behavioral indications of a deeper problem in the system. For me the key takeaway was that you should continuously monitor your processes as well as your systems in production (e.g. by gathering usage statistics) and fix the root cause when a problem is found.

Others

Here are some talks that I missed, but which received a lot of positive feedback. Thanks to ustream.tv they are available online. Also check out #CraftConf on Eventifier.

Jackstones: the journey to mastery

by Dan North (Slides | Video)

McDonalds, Six Sigma, and Offshore Outsourcing: Unexpected Sources of Insight

by Chad Fowler (Slides | Video)

Testing the Hard Stuff and Staying Sane

by John Hughes (Slides | Video)

The Better Parts

by Douglas Crockford (Slides | Video)

Functional Examples from Category Theory

by Alissa Pajer (Slides | Video)

 

Takeaways from Craft Conference 2014, Budapest – Day 2

I had the privilege of attending the speaker sessions of Craft Conference last week, the central theme of which was software craftsmanship. There were many inspiring talks and so was the venue. Think of it as TED for software developers. The icing on the cake was free beer, complimentary Palinka, unlimited coffee and a blues band. 20% of the speakers were women (but only 4% of the attendees) and ~350 of the 900+ attendees were from abroad. The event was virtually flawless. The usual systems that breakdown at scale: WiFi, food and toilets, all just worked. Plus there were small thoughtful touches such as English translations of useful Hungarian phrases on the attendee badge. Everything about the conference was impressive, including the raffle prizes which included a R/C drone!

Other interesting features were the use of Sli.do to manage audience questions in real-time, and a live-projected twitter feed. I had the opportunity to interact with practicing or aspiring software crafts[wo]men from Ukraine, Japan (ok, technically SF), Netherlands, Sweden, North America, UK and… India!

I’ve tried to distil out the summary of talks that interested me. They are ordered by my preference, not the order in which talks were actually conducted. These are my interpretations and my views, so please bear in mind that they could be wrong or biased. There were 3 tracks (parallel talks) over 2 days, so essentially I attended only about 1/3rd of the total. Day 2 was the first day of the speaker talks… I didn’t attend day 1, which was workshops.

I strongly recommend checking out the agenda and viewing the talks that interest you online, you might find some that I didn’t attend but are of direct interest to you. To me the top 3 recurring themes of the conference seemed to be:

1. State management in complex and distributed systems
2. Better automated testing & TDD
3. The abuse of agile (in development and in project management)

How I Learned to Stop Worrying and Love Flexible Scope

by Gojko Adzic (Slides | Video)

Probably one of the most-loved and honest talks at Craft. The storytelling was simply mind-blowing (an example of agile from 1628 AD, Ducati’s experience with the Second System Syndrome, threats to competitive advantage from Google & the Russians) and so was the message. I would highly recommend watching the video, Gojko is an outstanding speaker. The highlights were:

• The most common software development methodology these days is WaterScrumFall, in which all the essential planning is done upfront by management and the development is done in a [predictable] number of sprints
• This is because the concept of agile is not attractive to management, unless they truly believe in keeping scope flexible. Therefore agile generally remains underutilized.
• Agile is not just about continuous delivery, but also continuously reacting to local, temporal and human factors
• Project plans and roadmaps should not be linear, but literally a “map of roads”: multiple options with selection criteria (like a GPS). A roadmap with a pre-decided outcome is not a roadmap, but a road in a tunnel.
• Try new things, at a survivable scale and select the ones that work (Throwing away bad code is a way to reduce your technical debt)
• Don’t just ship software, make an impact
• On the topic of outsourcing: Usually the objective is to minimize costs, so the focus is not on excellence or flexible scope
• On User Stories:
  • Are a way to delegate details
  • Avoid translating the product roadmap into a set of JIRA stories, instead consider hierarchical backlogs
  • Add a victory condition to your user stories, which is related to changing user behavior rather than complying with the existing behavior (e.g. Monitor inventory *faster*)
  • A good user story is a survivable experiment

Recommended reading:

• Adapt: Why Success Always Starts With Failure, by Tim Harford (specifically the Palchinsky Principles)
• What Customers Really Want, by Scott McKain
• 50 Quick Ideas to Improve Your User Stories, by Gojko Adzic

Agility and the essence of software architecture

by Simon Brown (Slides | Video)

This was also one of my favorite talks. I like to think of it as Minimum Viable Architecture for developers. Simon is an inspiring speaker, more so because he eats his own dog food. The premise was simple:

• Agile delivery does not imply agile architecture
• Agile development still requires upfront thinking to define the overall architecture
• The team must have a shared, consistent vision of the significant structural elements of the product
  • With this pragmatic tradeoffs can be achieved: e.g. a monolithic deployment container, containing microservices
• UML was supposed to solve this problem, but it is poorly understood, not widely adopted and has its own overheads

The solution: NoUML! Abstraction is more important than notations. A team can define their own legend for these abstractions. Design diagrams are supposed to reduce complexity (through abstraction), not increase it. The 3 things that the team needs to have a common understanding of are:

• Structure
• Risks
• Vision

In real life, we rarely have 1:1 mapping between design diagrams and code. In general, a hierarchical C4 architecture diagram can solve this problem:

• Context
• Container
• Components
• And optionally, Classes

And yes, while good architecture is a shared responsibility, it is important that only one person (or a small group of people) are responsible for maintaining the overall architecture definition.

Recommended reading:

• Just Enough Software Architecture, by George Fairbanks
• Software Architecture for Developers, by Simon Brown
• CodingtheArchitecture.com
• Risk-storming

Getting Things Done at Scale

by Amber Case (Slides | Video)

This was a talk I could relate to a lot, because of GTD and the differences in corporate cultures of large, old organizations vs smaller, newer ones. Amber is a TED speaker, and it clearly shows. She shared her experiences during (and after) the acquisition of Geoloqi by ESRI.

Only 1 of 50 people she spoke to for advice had a happy acquisition experience, and the main reasons were:

Crippling management/overhead to get simple things done	Culture clash
Founder flight	Jealousy/blocking from parent company employees
Lack of detailed transition plan	Sprinters vs Marathon Runners
Loss of passion for original product	Loss of respect and cross-compromise

The solutions discussed were:

• Pre-negotiate, don’t be vague. Predetermine your outcome.
• Learn the local language (technical terms, tools, company culture)
• Win friends to influence people. Develop trust.
• “Beta test” people via small projects. Best code wins.
• Scale teams down from unmanageable numbers to 5-6 “doers”
• Communicate. Respect. Give first.
• Small revolutions
• Distribute stress

An interesting concept was the creative use of IRC bots, e.g. to send out a daily email summary of accomplishments (!done). Towards the end, she also shared an effective way of “organic hiring”: turning contractors into full-time employees.

It’s never too late to fight your legacy!

by Mate Nadansi (Slides | Video)

Mate delivered a very strong and reassuring message: Legacy code isn’t bad, just old or over-iterated… and, with some sensible planning, foresight, and many iterations of hard work, it can be replaced by more modern code. He explained how they achieved exactly that at ustream.tv. The presentation requires some familiarity with web programming but it would still benefit anyone working with legacy code… because maintaining legacy code builds character. 😎

Programming, Only Better

by Bodil Stokke (Slides | Video)

The core message was about how the introduction of multiple [mutable] states makes programs unmanageably complex. Topics included:

• Referential Transparency
• Representation of state using numbers instead of objects to it remains immutable
• “Encapsulated state” is still state
• Effect on concurrency
• Additional complexity added by control structures in contemporary program

The presentation itself was a work of art and for me the highlight was that Bodil was editing & running code from within the slides!

Recommended reading:

• Out of the Tar Pit, a 2006 paper about Functional Relational Programming by Ben Moseley and Peter Marks

Going Reactive: Event-Driven, Scalable, Resilient & Responsive Systems

by Jonas Boner (Slides | Video)

Jonas gave an inspiring and pragmatic talk about how the nature of, and expectations from, applications have changed dramatically over the years. The highlight was that not only did he distill them into the 4 attributes mentioned in the topic of the talk, but also shared valuable insights into how to practically achieve them from a technical standpoint. Some of the approaches discussed were:

Loose coupling Never blocking Asynchronous operations Actors and Agents Futures in Scala Minimizing contention

Maximizing locality of reference Single Writer Principle The relation between scaling up and scaling out Decoupling error handling from business logic Bulkheading to prevent cascading failures Maintaining consistent latency across blue skies conditions, high traffic and failure

Recommended reading:

• ReactiveStreams.org
• The Reactive Manifesto
• TypeSafe.com
• The Network is Reliable, by Aphyr

What Makes a Good Development Process?

by Bruce Eckel (Slides | Video)

Bruce Eckel is an industry veteran an author who needs no introduction (I read his textbooks in high school). I highly recommend going over his slides, they are very comprehensive, thoughtfully put together and highly informative. He has already distilled years of experience into a few pages, and I don’t think I can add any more value except quoting the one thing that stood out for me: The cognitive load of carrying tensions prevents us from doing creative tasks well.

Acknowledging CAP at the Root – in the Domain Model

by Eric Evans (Slides | Video)

This talk dealt with a very specific type of problem (CAP = Consistency, Availability & Partition tolerance), and solving it using a domain-driven programming model. One of the interesting concepts was that of defining aggregates within a distributed system. Instead of trying to keep the entire system in a consistent state, the contents of each aggregate are guaranteed to be consistent (even if equally stale). There is only a single point of reference into an aggregate. Aggregates are contained within a bounded context and transactional consistency must not cross these boundaries. Domain events interpreted within this context cause state changes. Eventually overall system consistency can be achieved through synchronization according to a reasonable SLA.

Don’t forget to check out Day 3!

Going Lean

If you haven’t noticed already, I’m an Entrepreneurship Enthusiast. I’m a big fan of small teams that get things done. I believe that no [tech] company should grow beyond 25 employees, and companies with more than 50 in the same location should be banned. I think that outsourcing everything other than your core competency is good, but outsourcing is a bad idea when your sole motivation is to cut costs, rather than to be able to focus on your core competency. In the long term, everybody walks away dissatisfied from such an arrangement.

Intrapreneurship can enable innovative teams to thrive in large organizations. Unfortunately middle management (in India) is paranoid, unwilling to take risks and sometimes just plain clueless, so innovation often ends up being a forced phenomenon. (Although like in everything else in life, there are exceptions.)

Earlier this year unrelated circumstances forced me to part ways with my employer in a hurry. While looking around for new opportunities, I met Vinit who suggested that I attend the upcoming Lean Startup Machine event to meet new people and get new ideas. And boy, was that good advice.

I showed up at LSM with no expectations, nothing but a laptop and an open mind. When I learnt that attendees could pitch, I made the first one. 13 teams eventually self-organized out of the 64 attendees and mine turned out to be one of the top voted ideas. I was lucky to have an investment banker, a web entrepreneur, another lean entrepreneur and a CTO volunteer to join my team — and having that variety of expertise & experience in the team made all the difference.

We quickly bonded as a team and with the help of LSM Mentors we slowly but surely implemented lean principles in validating our idea. By day 2 we had a good idea of what we wanted to build, and by day 3 of pivoting and validation we even had several Letters of Intent from potential customers. Eventually we went on to win the competition and as a result SafeRoads India was born.

LSM was an eye opener and a fantastic learning experience, some of which I’d like to consolidate here. The advice and learning extended beyond the 3 days of the workshop into the followup sessions with Investors and Mentors (our prize!), and the new friendships that were forged amongst kindred souls.

_________________

Inspiration started with Drew Nagda, Director of Corporate Partnerships @lean narrating his story and how not to underestimate the potential of an idea to make you money. He went on to explain some of the core fundamentals related to Lean Startups:

• Defining “customer”, “problem statement” & “solution”
• Customer validation (using the weakest acceptable outcome as the minimum success criterion)
• Pivots (changes in strategy without changing vision)
• MVP / MVE (Minimum Viable Product / Experiment)
• The cycle of Hypothesize (riskiest assumption) -> Experiment -> Validate (get out of the building) -> Analyze
• Exploration (finding the problem), Pitch (collecting “currency”) and Concierge (meeting expectations)

_________________

For me the highlight of the event was the talk by Mukund Mohan, CEO of Microsoft Accelerator. It has been the most inspiring and practical advice I have ever come across, especially his quote that I have taped on to my desk: “My discipline will beat your intellect”. I have since been recommending Mukund’s Blog to everyone I meet. Mukund initially touched upon what they look for in the MS India Accelerator program, mainly a disruptive product idea, large potential customer base (interestingly only  1/10th of what they look for in the US market) and “coachable” entrepreneurs. He then talked about the outstanding benefits of being part of the accelerator, and trust me, they are very nice indeed because their primary objective is to engage and grow the startup community rather than make money off of it (Though I suppose that would be a secondary effect in the long run :-)). Towards the end he spoke about the 3 things he looks for in entrepreneurs:

1. Attitude — Never giving up, Being an opportunist.

2. Discipline — Because success gets money, not the other way round (a.k.a. “Success is determined by revenue, not funding”)

3. Storytelling — Painting a convincing picture and envisioning the dream for all three: Investors, Customers and Hires.

_________________

A close second was a mind-blowing talk by Sridhar Ranganathan, in which he dived into the philosophy of what is a Minimum Viable Product and reiterated my long-standing suspicion that in any project, only the top 10% items actually get done.

_________________

The next day, Prof. Nandini Vaidyanathan (CARMa Connect, author of Entrepedia) took us back to the basics of “social” (share, connect, collaborate, create). She shared some very interesting anecdotes, such as the one about the enterprising kid who was selling local chai to people waiting in line when Starbucks opened in India, about how eBags’ VP responded personally to a customer’s blog post about poor quality zippers — resulting in a change in their supplier and turning an angry customer into a devoted evangelist, and how 95% of Lego’s products are designed by their customers.

_________________

Haja Sheriff from Microsoft shared his story about making Microsoft a resounding success with Universities & Colleges in India, by applying lean thinking to change the focus from the perceived “market share” problem to the actual problem, which was to achieve increased employability for students.

_________________

That evening, Prasanna Raghavendra of CloudMunch shared his insights about what not to do in lean product development. Some of them were:

Build

• Don’t build from scratch (reuse)
• Don’t go back to the drawing board too often (build and iterate)
• Don’t try to do everything yourself (focus on the best use of your time, for everything else find someone else or outsource)
• Don’t stop refactoring (change is good)

Deploy

• Don’t architect for depth (YAGNI)
• Don’t solicit unused feedback (everyone will give feedback but focus on the meaningful ones)
• Don’t formalize contribution within the team (can lead to imbalance and cohesiveness issues)
• Don’t ignore the ecosystem (build what is relevant)

Manage

• Try to build software that “auto-tunes” itself: self run, self service and self elimination of problems

_________________

On the last day, Deepak Shenoy shared his wisdom (learnt the hard way) about the 5 myths of being a financially lean startup, which was a big source of help for everyone in the room — because in a world full of people with ideas, it is execution that makes all the difference. Unfortunately due to communication gap he didn’t have enough time to complete his presentation, but I would strongly urge you to read about it here.

_________________

LSM was a very encouraging and enlightening experience. I would strongly recommend it to anyone who has a voice in their head and is trying to figure out whether it is crazy or not. Even if you don’t have a startup idea yourself today, it is a fantastic place to meet new people, learn new things and get new insights — maybe even play a role in bringing someone else’s idea to life. And it doesn’t have to be pure technology, ideas pitched at LSM included a Marriage Portal for Bollywood style weddings, a chain of healthy takeaway breakfast stores and a hotline for quick medical advice. So go on, get out of the building!

Happy New Year MMXI!

Happy New Year! If the 2012 Phenomenon is to be believed, then we have about 01 year, 11 months and 01 days before the world reboots. Must make the most of it.

On the brighter side, though:

 

In the last week, I have made substantial changes to the categories on the left, just to make them less cluttered and more meaningful. My apologies if this broke something that you had linked to. This was long overdue and any future changes would be relatively insignificant.

Live long and prosper, and follow me on twitter: @survivalcrziest.

 

Flight Simulators: How Humans can no Longer Tell the Difference Between Reality & Illusion

That was the title of my session at Barcamp Bangalore 9. It was more of a presentation actually, since I was talking about a topic that not many people know about (at least not in what was going to be my target audience that day). In fact, that was the idea behind the whole thing: to raise awareness about how advanced & mature simulation technology is today, where it is going, and where lie the obstacles and opportunities. I also included a bit of philosophy and some high-res photos. Here’s the synopsis of my session:

Since you are in some way associated with technology, you are probably aware that technology is/can be used to train humans for dangerous, skilled and expensive-to-repeat tasks, such as Driving, Surgeries, Operating specialized equipment, and well, flying.

– Did you know that there is something known as a “Zero Flight Time” Simulator, which is so realistic, that every hour that a potential pilot spends in it, is considered equivalent to flying an actual aircraft? And that pilots go flying on an real, revenue-generating trip for the very first flight in that type of aircraft without having ever set foot in it before?
– That there are only a handful of companies in the world that have the technological capability to create the accurate reproduction of reality (sight, sound, motion & vibration) that makes this possible?
– That Simulation is probably the only field of Computer Science, that makes use of just about everything that Computer Science is made up of, from Algorithms to 3D Graphics, from Embedded to Web, from Usability to Networking?

This discussion is going to be about Simulation (mostly Flight Simulation) and some of the amazing technology that lies behind it. How the Simulation, Gaming & Movie industries are feeding each other, and feeding off each other. And finally, Simulationism: the possibility that you and I are complex objects living inside a Computer Simulation.

Here’s a stripped-down, text-only PDF version (77 KB) of all the PPT-fu that I did (Sorry about the crazy contrasts, they make more sense with the pictures behind them). Do leave a comment letting me know your thoughts.