An Exercise in Analytics: Using Microsoft SandDance to Visualize Trends at CraftConf

I’ve been lucky to attend every CraftConf Budapest since its inception in 2014.  It has always been a mind-expanding experience, with a healthy mix of established and newcomer speakers from the tech industry worldwide. Although Craft, as the name suggests, is focused on Software Craftsmanship, the specific topics of talks vary each year as industry trends fluctuate. I was interested in taking a deeper look at trends of technologies and paradigms as they become more or less popular over time, and to find those, if any, whose popularity has remained more or less constant.

While doing some research into data analytics for a project at work, I came across Microsoft SandDance. My interest in the CraftConf data was the perfect opportunity to teach myself SandDance (and some Python as well). So I put together a quick experiment: Using Martin’s excellent tutorial on Web Scraping with Python as a reference, I wrote  a Python script that scrapes CraftConf’s talk descriptions (using archives from 2014, 2015 and 2016) and produces a CSV file of the most frequently occurring words. Obviously standard English language words like pronouns, days of the week, and so on are ignored. What we’re left with is a list of top 100 words for each year, and their frequencies of occurrence, which can be visualized  in SandDance.

Although this approach is quick and more or less effective, the limitation is that it may not accurately reflect trends for phrases like “Agile Methodology” — the word frequency of “agile” may not be the same as that of “methodology”. But that’s something that can be worked on later. So although I would take this as a good indicator (which meets my purpose), I wouldn’t use the analysis outcomes as a serious reference.

What the Data Tells Us

Here are some interesting findings from the first pass (150 of the most frequent words selected, out of those 100 produced after filtering out common and punctuated words):

  • “Product” showed up 26 times in 2015 and nearly doubled to 51 in 2016 (a growing trend: no talk, some talk, twice the talk…)
  • “Functional” [programming] appeared 16 times in 2014, and not in the other 2 years (something that’s coming and going?)
  • Similarly, “Architecture” showed up respectively 15, 37 and 23 times (up and down)
  • “DevOps” was an equally hot trend in 2014 and 2015 but didn’t show up in 2016 (presumably because the hype is over)
  • “Microservices” appears 29 times in 2016, but didn’t show up in the previous years (so there is a recent spike in popularity)

 

CraftConf2016

Tim Steigert’s Closing Keynote at CraftConf 2016

See for Yourself

As a fun exercise in data visualization and trend analysis, I encourage you to try it out for yourself, using the CSV file produced by my script. To start with:

  • Load the dataset: Dataset > Web > CSV file (Keep “First line is header” checked)
  • Set the URL to the CSV file link above, and click Load
  • View as: Column
  • X Axis: Keyword
  • Sum by, Facet by: None
  • Color by: Keyword
  • Sort by: Frequency
  • Set the X axis bins to: 100

The typical way to drill down using SandDance would be:

  • Select a keyword (say “lean”)
  • Click Isolate. Everything else gets filtered out (note the “Filtered” count increased from 0 to 2)
  • Now you can check “Details”, or “Facet by…”, for example
  • To go back, simply click “Filtered” to clear the selection
CraftConfSanddanceArchAnalysis

Isolating the keyword “Architecture” in SandDance

Isolating the “Other” keyword will reveal a whole lot of keywords that don’t show up in the first 100 bins. You can also take the SandDance tour (by clicking on Tour) and discover many other interesting ways of playing with SandDance.

You can find the source code in my Git Repository WordFreqCount. If you find it useful, please feel free to reuse, derive from or improve it. Note that credit goes to Martin for the original code on web scraping using Python and Beautiful Soup, which I largely adapted from. And of course, thanks to Microsoft for making the elegant and powerful SandDance available for free!

Amuse UX Conference, Budapest

Last week, I had the privilege of being part of a group attending the first edition of AmuseConf on behalf of our company. Amuse is “an international conference for anyone interested in how to design and develop successful products that users love”. It’s organized by the same good folks that bring us the outstanding CraftConf year after year, sponsored primarily by Prezi and UStream (and SAP in case of Amuse). They did a near-perfect job, with only minor glitches with the seating and catering on the first day. Considering that the Big Data oriented CrunchConf was also literally next door, the event was practically flawless. Fast, uninterrupted WiFi and no food options for vegetarians/vegans remained a hallmark this organizing team (even though Tom Illmensee, event MC is himself vegetarian 😉 ).

(BTW, if you’re wondering why so many tech conferences are being hosted in Budapest, the event’s WiFi password should give you a hint):

20151103_083708867_iOS

510 attendees from 32 countries (as far away as Australia) made Amuse a roaring success, as did its impressive lineup of Speakers27% of the speakers were women, which is great for a tech conference — I hope next year we have even more!

Below is a summary of the talks I found the most relevant to my work. But by no means does that mean you should skip the other talks… depending on where you are and what you’re doing, you might be interested in some of the eclectic topics covered such as:

  • Designing web interfaces for children by Trine Falbe
  • Conducting research outside “sample of convenience” by Bill Selman from Mozilla Foundation
  • Design Thinking by Tobias Haug of SAP (my favorite quote: “Innovation = Execution x Creativity”)
  • How to get your dream UX job by Andrew Doherty of Google (worth checking out just for his mad presentation skills)
  • The Ethical Designer by Cennydd Bowles
  • Storytelling in a multidevice landscape by Anna Dahlstrom

Design Equilibrium

By Jonathan Lupo

Jonathan opened the conference with a very engaging talk drawing parallels between businesses and ecosystems: a “balanced exchange of value between Actors, Enterprise and Brand”. He gave practical examples citing the application of Lynn Shostack’s work on Service Blueprinting to a transformation in the healthcare industry. I strongly encourage viewing his inspiring talk on YouTube.

His core suggestion is a separation of Product Design from Service Design. The latter “fills in whitespaces between points of [rich] engagement provided by products”, helping to restore balance to the overall experience, and hence the business ecosystem. This is the real intangible value of services, as opposed to products.

He also proposed the concept of an “Engagement Model”: a framework to contextualize all the data a business collects.

UX: Design as a Science

By Joel Marsh, author of the UX Crash Course

Joel’s key message was that “Scientific UX Design is reproducible”: essentially drawing on the principles of the Lean Startup and applying them to the UX domain. His presentation was one of the most popular and engaging ones, and his quotes and examples garnered a lot tons of positive feedback. One thing that struck me was his exposition on the two types of creativity: Creative expression and creative problem solving. He noted that an over-applicability of creative expression can make you feel good as a designer, but result in an over-designed and bloated product:

ArtVsDesign

Another talk I would highly recommend watching when it comes out on UStream.tv.

Making Dog Food a Part of Your Balanced Diet

By Toby Sterrett

Toby used his work at Simple Bank to highlight the pros and cons of “eating your own dogfood”. The initial employees used the app themselves, and one of the downsides was that the missed revelation that users of such a smooth app had to deal with a paper form-based process to close their account, which took up to 20 days.

Another inspiring talk that you should definitely check out, full of quotes of wisdom like:

  • “Delight is design’s superpower”
  • A past discussion on Leadership strategy: “Build a shared vision, get the **** out of the way”
  • “UX is not about throwing technology at a problem, but throwing people at a problem”

On the other hand, Simple A/B tested as many as 16 variations of their login page (for more examples, check out UserOnboard.com).

Live posters being created by @remarker_eu

Live posters being created by @remarker_eu

How We Built Hotjar and Onboarded 50k Users in a Year

By Dr. David Darmanin

David used practical examples from Hotjar to support his model of “Drivers, Barriers and Hooks” when dealing with site visitors. He also put a quirky twist on some timeless wisdom:

The two most amazing insights for me were:

  • Hotjar captures every single customer interaction on a Trello board, and uses that feedback to prioritize their features.
    • They also make their roadmap public, which demonstrates their commitment and at the same time reduces enquiries about feature requests
  • They use the income from their paid customers to fund the creative freedom to build features for their free customers

The Invisible Interface: Designing the Screenless Experience

By Avi Itzkovich

Avi, founder of UXSalon, opened with a discourse on recent editions of Microsoft Productivity Future Vision. From there he led the discussion on towards a future without bigger and wider screens (which wouldn’t require “superhuman arm strength”):

  • “The most profound technologies are the ones that become invisible” 
    • Like automatically opening sliding doors
  • “Voice UI is the future”
  • “Gesture control is here to stay, but not on screens”

The Best Interface is No Interface

By Golden Krishna

Golden surmised that we are all “chipping away at digital chores”, and we don’t have to be “slaves to screens”. He has laid the foundations of the #NoUI movement with his book“The Best Interface is No Interface”. His excellent talk (slides here) was supported by book reading and real examples. Also, don’t forget to check out his accompanying toolkit on “how to create elegant solutions with no screens”.

For further inspiration to join the movement,  take a look at his Producthunt collection of “interfaces that require little or no time with screens”.

Magical UX and the Internet of Things

By Josh Clark

Josh opened with an announcement of his book release: “Designing for Touch”. His presentation was literally magical, complete with a wand, to the point that he managed to tie in together excerpts from preceding talks and put the whole conference in perspective. I found a similar slidedeck from one of his previous talks here, and I highly recommend taking a look at it while we wait for the official conference videos to come up on Ustream.tv. It was a treasure trove of out-of-the box examples like:

  •  Augmented REality Sandtable (ARES), which literally turns dirt into a high-tech, military-grade user interface… using not much more than a Kinect and projector
  • Grab Magic, which brings superpowers to data transfer
  • Propeller Health, which connects Asthma inhalers to phones for health monitoring

Josh’s key message was “interaction at the point of inspiration”: that we should think of “the whole world is an interface, just like it has always been”. He proposed “thereables” instead of wearables: bits of smart technology in the physical space where we would expect to interact with them, not something we burden ourselves by carrying or wearing all day long. To this end, he suggested that “the smartphone is Magic Wand 1.0 for everyone” and we should start thinking of it as just more than a screen.

IMG_4185

Regarding user interfaces, he had 3 bits of advice that I found remarkable:

  • “Technology should amplify our humanity”
  • “We shouldn’t educate users on how technology works, unless we really *have* to”
  • “Honor intention, don’t assume it”

Josh ended with a call to action:

Like this one.

 

 

 

 

 

FilesDesk

2012

This started out as a side project between a couple of us at work to sharpen our C#/XML skills and generally do something challenging. A C#.NET Application + Service that enables browsing files on disk by Tag (as opposed to Path). Typical usage:

  • A Tag Cloud for your hard disk
  • Add/remove tag(s) to selected file(s)
  • List & Filter files by tag(s)
  • Monitor specified folders (ex: My Documents) for new/deleted/renamed files
  • Portable XML Database

Useful for Researchers, Writers and Obsessive-Compulsive File Hoarders.

DOPE: Distributed OPerating Environment

November 2003

I did this project with a friend around that time that Grid Computing was getting popular in the mainstream with SETI@Home (in its pre-BOINC avatar).

A Client-Server Java app that used Distributed Computing principles to:
a) Break down a given, computationally intensive task into chunks*
b) Distribute these chunks to clients on the network, favouring those with lower latency
c) Re-assemble the results returned by clients into a single unit
d) Re-send chunks for which result was not received within stipulated time

*Limitations apply: this was not a general-purpose implementation

The Art of Listing

As I had posted here in 2007, I have been trying to make an effort towards paperless organization of my lists, most of which are ToDo items. The Palm device that I was originally attempting to use for this effort turned out to be a headache because of limitations on formats, storage capacity, speed, interoperability and expandability. I ended up giving it away to my cousin brother who is a student, to make his first attempt at getting organized 🙂 In the meantime, I picked up a Sony Ericsson P990i, which let me do a lot more, faster and more efficiently (Of course, that device is also fast approaching its event horizon). I found that I have so much going on in my head that often it was a pain to take out the phone, flip it open, navigate to “Tasks” or “Notes” and start typing. Going 100% paperless wasn’t working out too well, sometimes during this physical process I would lose track of my mental process (i.e. forget the idea or task that I wanted to note down). Over the years, I have arrived at the following hybrid approach, which helps me get things done effectively:

1. On my device, I maintain the following lists, in the following order, each of them almost like a Product Backlog:

1) This Week 5) Online – Stuff to do the next time I’m in front of a computer, like e-mailing somebody
2) Weekend 6) Projects – Not just software, even real-world projects like scale models
3) Next Week 7) This Quarter
4) This Month 8) This Year

Plus, the following “dynamic” lists:
a) Groceries – Since the stuff I buy every week/month is almost always the same, I just have a master list in which I keep moving things between “Pending” (unchecked) and “Completed” (checked) depending on what I run out of
b) Shopping – Other things to buy next time I’m out
c) Travel – Places to travel to on the weekends
d) Focus – 1 to 5 items I’m currently focusing on (e.g. “Get to work on time” 🙂 ), to keep reminding myself regularly

2. My phone lets me prioritize tasks within each list, on a scale of 1 – 3. Also, for example, within “This Month”, if “Pay Rent” has been completed, it gets checked into “Completed” and doesn’t get deleted. At the beginning on next month, I simply uncheck everything back into “Pending”.

3. I maintain a single sheet of pocket notebook-sized paper (more if I’m actively noting down ideas/tasks for an ongoing activity/project), akin to a Sprint Backlog, with the following:

Front Side Back Side
Today – Things to do today (mostly at work) This Week – Including weekend commitments
Calls – Phone calls to make + e-mails to send Home – Things to do when I get back from work

4. Every weekend I move stuff from the “Product Backlog” (long-term list of stuff on the phone) to the “Sprint Backlog” (short-term list of stuff on paper), and *wait for it* stuff gets done! I never use more than one sheet of pocket notebook-sized paper in a week, and this way I also always have paper handy to quickly note down stuff (on the margins). Finally, in case I ever lose/damage my phone (which is backed up every 2 weeks), I don’t lose the things I had planned for the week.

Am I going overboard? (After all, it’s just a glorified ToDo list.) I don’t think so. I find that by keeping things prioritized and focused this way:

1.  I manage to get a lot more done without worrying about what I’m forgetting to do.

2. I don’t lose track of things that I would eventually like to do, but don’t have the time for right now (or this week, or this month, …)

3. Moving the prioritization and organization out of my head helps me think clearer and focus 100% on the task at hand.

But it doesn’t end there. Over a period of time (and with a lot of self-imposed discipline, I must add), I have managed to harmonize the short-term (a.k.a. “sort it when you see it”) organization of things that I come across everyday. I do this by managing the following “tags” (often as Folders, in some cases even physical file folders) across my Inboxes, Browser Bookmarks, Hard Disks and scattered notes (including those on my phone):

  • BlogThis
  • ReadThis
  • WatchThis
  • DownloadThis
  • FollowUp

I visit these as and when I have the time and keep emptying them out. With the addition of lists (as notes) for Movies to watch, Music to get, Books to read and Scale Models to buy, my little universe of lists is complete!

Stuff that I learned along the way, though:

1. Hybrid is more practical than paperless.

2. We need a device (implant?) that can make a note when the wearer thinks of it (and where to put it). The interface & actions between thought and task noted are, well, so ’90s! (Note: Speech Recognition is also so ’90s)

3. It’s best to stick to simple formats like Text and CSV instead of proprietary ones (like Excel). Simpler formats are easily portable and retrievable in case of failure, and suffice for making lists. If your list seems to require a complicated format, well, simplify your list!

4. It may be a good idea to reuse Visiting Cards and such, but your handwriting needs to be tiny.

5. Evernote can probably help.

UPDATE: [2011-07-16] I have since migrated all my lists to my new BlackBerry Curve 9300.

UPDATE: [2012-03-09] I discovered Todoist, which is quite simply the Tao of using Computer Science to solve problems. Although using it means that my todo list is now in the cloud, something that I’m [still] not very comfortable with, I find it indispensable to manage long-term projects. I initially found it attractive due to its Outlook integration, which meant that I didn’t have to grapple with numerous tasks disguised as emails, but the app is constantly being improved with new features, like @labels that enable a task to be present in multiple lists. HTML5 support means my list is now available offline, and it syncs effortlessly across devices, including my BlackBerry. I occasionally take local backups with Todoist Backup.

UPDATE: [2012-09-15] With my mind emptied of the long term stuff now safe on Todoist, I have started relying more on my memory for day-to-day things. I’m also trying to do less and focus more on the important things (not to mention years of long hours have significantly shortened my “backlogs”), and try not to take on more than I can comfortably remember over the span of a few days at a time.

UPDATE: [2012-10-05] I have a new revived obsession with Whiteboards at home. I’m trying to keep it under check to avoid looking like too much of a mad scientist…

UPDATE: [2013-07-12] Since Todoist Backup no longer works and I don’t have the time to figure out what changed, I have upgraded to Todoist Premium to be able to make use of their backup service (amongst other cool features). I now exclusively use Todoist coupled with a one-page note on my phone, which has tasks for the day to week range, plus trivial items (< 2 min) that don’t need to go into Todoist. Also, the other day I found the phone I used to use before my Palm device: Nokia 6820.

Usability, UX Resources

Chennai-headquartered CHI-SI is a chapter of Association for Computing Machinery‘s (ACM) special interest group on Computer-Human Interaction, SIGCHI. CHI-SI hosts a national conference every year.

Bill Scott, Director UI Engineering, Netflix, puts down his musings on rich web design and user interface engineering at Looks Good Works Well.

SF-based Adaptive Path does some really interesting work on user experiences. They also maintain an interesting blog.

GUI, HCI, HFE, MMI

Interfaces, interfaces, interfaces. That’s what this section is all about. You may have access to the most powerful computer on Earth, but it is of little use if it doesn’t provide you an interface that lets you tell it what exactly you want it to do. In everyday life, we interact with many, many machines through user interfaces – not just computers and cellphones, but even alarm clocks, cars and microwave ovens.

User interfaces are as much of a psychology problem as a computer science problem, maybe even more.

I believe that user interfaces have an immense potential in the way we write computer programs. I’m talking about programs writing programs. We’ve come a long way since writing arcane commands on interpreter prompts but the more user-friendly programming interfaces become, the more people can translate their ideas into actions.

Finally, most computer interfaces today are 2-dimensional. In all probability, this is merely hereditary. Thinking “out of the box” requires a box, and boxes are 3-dimensional.

The following fields are closely interrelated:

  • GUI (Graphical User Interface)
  • HCI (Human-Computer Interaction, also referred to as CHI)
  • HFE (Human Factors Engineering)
  • MMI (Man-Machine Interface)

UPDATE: [2011-01-18] Rehashed this section and added:

Related:

References:

  • Handbook for Human Computer Interaction (2nd Edition), By Andrew Sears and Julie A. Jacko (CRC Press, 2007)