Paper Reading #4: Gestalt: integrated support for implementation and analysis in machine learning

References:
Gestalt: integrated support for implementation and analysis in machine learning by Kayur Patel, Naomi Bancroft, Steven M. Drucker, James Fogarty, Andrew J. Ko, James A. Landay.  Published in the UIST 2010 Proceedings of the 23 Annual ACM symposium on user interface software and technology.
Author Bios:Kayur Patel is currently pursuing a PhD in Computer Science from Washington University.  His studies focus on machine learning algorithms where behavior is learned from data rather than specified in code.  Naomi Bancroft recently graduated from Washington University with degrees in Computer Science and Linguistics.  She is currently employed by Google in Mountain View, CA.  Steven Drucker is a principal researcher at Microsoft Research focusing on human computer interaction when dealing with large amounts of information.  He holds a PhD from the Computer Graphics and Animation group from MIT Media Labs.  James Fogarty is an assistant professor of computer science and engineering at the University of Washington.  He holds a PhD from Human-Computer Interaction institute at Carnegie Mellon University.  Andrew Ko is an assistant professor at the information school at the University of Washington.  The goal of his research is for software evolution to be driven more by human need and less by technological constraints.  James Landay is a professor of computer science and engineering at the University of Washington.  He holds a PhD in Computer Science from Carnegie Mellon University.
Summary:
Hypothesis
The authors hypothesized that a structured representation would be most useful when developers first started a project.  They wished to show that it can significantly improve the ability of developers to find and fix bugs in machine learning systems.
Methods
The participants were asked to use a general purpose development environment to create, edit, and execute scripts.  They were given an API that could be used to reproduce all of Gestalt's visualizations.  The baseline condition and Gestalt used the same data table structure to store data, but in the baseline participants had to write code to connect data, attributes, and classifications.  Participants were asked to build solutions for two problems, sentiment analysis and gesture recognition.  The solutions had five bugs built into them and the participants were supposed to locate and fix as many bugs as possible within an hour.
Results
Participants unanimously preferred using Gestalt to the baseline, and were able to find and fix more bugs using Gestalt.  The authors noted a marginal effect of trial on the number of bugs found and fixed, as participants found more in the second trial.
Discussion:
I found the paper to be very effective at communicating what it set out to demonstrate, and intriguing as well.  A good development environment and effective tools can make a huge difference in the efficiency of writing and debugging code, and I think that it was interesting for the authors to target a more effective process rather than just focusing on the output.  There were limitations in the study, but the authors made a point of listing out some of the biggest ones at the conclusion of the article.  In particular, I noticed that although they state that a structured representation is probably more useful when developers first start a project, their study consisted of putting people in front of code that had already been written.  While not exactly counter-productive to their argument, it did lead a little bit away from the original path.