In recent years, deep learning models have gained a lot of attention. In our formative study, we found that most developers use online search, such as tutorials, blogs, and official documents, to find example models. However, they often had a difficult time choosing appropriate model structures and hyperparameter values. They expressed their needs to understand the network structures and hyperparameters used by other developers on related tasks and datasets.
In this work, we present ExampleNet, a novel interactive visualization interface for exploring common and uncommon design choices, such as neural network architectures and hyperparameters, in a large collection of open-source deep learning projects.
Many neural network visualization tools have been proposed to support different activities in neural network development, such as TensorBoard, TensorFlow Playground, LSTMVis, etc. ExampleNet differs from these visualization tools in three perspectives:
We conducted a 45-min semi-structured interview with 10 participants to understand the difficulties they faced and information needs they intended to discover when searching and designing the neural networks. These participants are all computer science or data science students that are recruited from Harvard University.
During the interview, we encouraged participants to contextualize their answers based on their recent experience of learning and building deep learning models. I coded participants’ responses to each question and then categorized them following the card sorting method.
We found that programmers often search and adapt neural network examples on GitHub rather than building a neural network from scratch.
Nine of ten participants said the first thing they would do was to search for GitHub projects that perform similar tasks on similar datasets.
I list ten information needs our participants wished to discover from GitHub examples when designing neural networks.
Participants also told us about the difficulties of discovering those information needs:
During the formative study, I also showed participants some mockups of neural network comparison tool. I collected their feedback using card sorting method.
I summarized three design principles based on the information needs.
In our interface, we have four main features:
I developed a semi-automated data curation process that extracts model characteristics from GitHub projects.
I conducted a within-subjects study with sixteen participants to evaluate whether ExampleNet could help them more effectively develop an awareness of design choices available to them when designing a deep neural network.
I selected two common deep learning tasks: image classification and text classification. In the control condition, participants were allowed to use any search engines they were comfortable with to find online resources. In the experiment condition, participants were only allowed to use ExampleNet without any access to other online resources. For each task, participants were asked to design neural network architectures and hyperparameter settings.
After each task, participants were asked to rate the cognitive load of the assigned task. After finishing both tasks, participants were asked to fill out another survey to directly compare the Online Search and ExampleNet conditions.
We recruited sixteen master students in Computer Science and Engineering or Data Science at Harvard University. And they had diverse expertise in deep learning.
Each study took about 70 minutes. Each participant was given 20 minutes to finish each task.
We also assessed the models they designed. We found that using ExampleNet significantly reduced participants’ mistakes in neural network design and hyperparameter design.
I carefully collected and classified each participant's feedback. Their feedback shows that participants felt more confident about the neural network they designed with ExampleNet and rated it as more helpful than Online Search when designing neural networks. They also felt less mental demand, hurry, and frustration when using ExampleNet than online search.