Process of prototypes
To tackle the design scenarios, this project made use of multiple research and design methods. The main paradigm used was research through design, which resulted in many phases, prototypes and tests. This chapter explains how the paradigm was implemented and which phases it resulted in. The prototypes belonging to the different phases are briefly shown and finally testing methods are shortly addressed.
Research through design
Research through design (RTD) allows designers to engage with wicked problems that cannot be easily addressed through science and engineering methods (Zimmerman, Forlizzi, & Evenson, 2007). Designs are researched to improve the designs, which are then used to conduct better research, see figure 1. The designs are prototypes which act as hypotheses and are tested in their respective environments. Research in context is also being called for in existing smart home literature: “Studying technology in a representative context of use will be crucial to assessing its suitability for everyday use and whether or not it addresses inhabitants’ intended goals.” (Mennicken et al., 2014)
A schematic showing the actions and proceeds of RTD.
Pivoting with research through design
Because this project started with a broad scope, it needed a method to narrow this scope as the project continued. Instead of making unfounded decisions at the start of the project, the choices were developed as explorations continued. Three moments were planned where the project could pivot into another direction. As new knowledge was uncovered, fruitful directions could be selected. This resulted in a narrow scope at the end of the project, forged from research knowledge.
Phases and pivots
This project was divided into four phases, see figure 2. The first phase was used to create an understanding of the context and to approach users. The following two phases focused on designing an experience for the nonenthusiast and the last phase focused on the enthusiast experience.
These phases made use of differing exploration and validation techniques. The first phase used literature research, interviews and a contextmapping session. The following three phases made use of prototypes for explorations and tests for validation. The first phase and its results have already been discussed in previous chapters, thus these chapters discuss the prototypes and tests.
Overview of phases, methods and participants.
Phase 1: a minimum level of control for non-enthusiasts
This phase focused on solving the reading light scenario for the nonenthusiast. The prototypes developed during this phase made use of the opportunities found in the context analysis. The prototypes in this phase started out with novel principles, see figure 3. Quickly it became clear that users need controls they recognize, both in location and usage. Figure 4 and 5 show the following steps to create control for a non-enthusiast.
Scenario 1, it allowed participants to undo the latest change of the lighting. They were meant to use it when it got too dark to read.
Scenario 2, it was placed next to a light and allowed participants to dim that light. The prototype could also toggle automations on and off.
Scenario 3, it allowed participants to control a light and disable the automations automatically. To enable automations again, the dedicated button had to be pressed.
Phase 2: usability and accessibility for non-enthusiasts
After the previous phase came close to solving the reading light scenario, the project pivoted to refine this design. The requirements for the device were re-evaluated, which resulted in new designs (figure 6 and 7). To evaluate how well participants could understand these designs, they were validated with usability tests. This isolated small changes, making their effect more apparent. After two usability iterations, these changes were put to the test in a scenario test again (figure 8).
Usability 1, a new form factor with different ways to indicate functionality. These indicators were tested against each other, this was the winner.
Usability 2, this prototype reused an old prototype to test the new interface. This was also the point at which the indicators were animated.
Scenario 4, after usability testing the design was tested in the scenario again. This was a new prototype that embodied all the previous learnings.
Phase 3: communicating use and responsibilities to enthusiasts
With the experience of the non-enthusiast validated, it was time to focus on the enthusiast. They had to learn about a complex product, be able to configure it and be aware of their responsibilities. To achieve this, an onboarding app was designed. This onboarding was tested and iterated on, first without context and finally with context. Figure 9 shows the final onboarding prototype and figure 10 shows the final physical prototype. Try the onboarding prototype here (works in chrome): graduation.emilflach.com/config.
Communication 2, succinct descriptions and targeted information for enthusiasts to understand usage and configuration.
Communication 2, the final physical prototype used with enthusiasts.
Participants
For all these tests, the participants were sourced at the IDE faculty. Using participants from the faculty allowed for weekly tests. These participants did not have to be arranged before the actual testing day. The tests also frequently made use of a testing environment which was at the IDE faculty.
Non-enthusiasts
Using design students does influence their knowledge of testing methodology, as this is part of their studies. This is a limitation of this method and might have skewed results more positive than they would be in a representative participant group. The testing protocols did make sure to provide the participants with very little information, similar to a first time non-enthusiast user.
Tests completed with the scenario method had 6 participants per test. The usability tests had 10 and 8 participants. All tests were completed in one afternoon, limiting the number of participants. The tests in scenarios generally took longer, thus fewer participants were involved.
Enthusiasts
Again, using design students influences their knowledge of testing methodologies and might have skewed results. To make sure the design students approached the knowledge of an enthusiast, they were given a lot of information about the system. The design students were possibly more akin to an enthusiast than a non-enthusiast, as they are technically savvy.
For the final validation, the participants were specifically selected to be technology enthusiasts. This was done to improve the representability of the participant group. Both tests had 6 participants.
Scenario test method
All scenario tests were done in a similar manner. Between tests, the methodologies improved, but the outlines stayed the same. For detailed methods and insights per test see appendix E.
These tests made use of the living lab at the Industrial Design faculty in Delft, see figure 11 and 12. The lab is a living room specifically made for testing in a living room situation. This room was therefore perfect for testing with the reading light scenario.
The living room used as a test setup.
A participant reading in the test environment.
Usability test method
The usability tests called for a more sterile environment with less changing variables. This allowed for tests that compared designs or test comprehension from just interface feedback. The details of these tests can be found in appendix F.
These tests were conducted in the main hall of the IDE faculty. During usability 2, participants were positioned directly across the researcher, such that the hands of the participants could be recorded.
The participants sat straight in front of the researcher, which allowed the hands to be recorded as well as the audio.
This is what the recording looked like, which gave a better understanding of what the participant had difficulties with.
Communication test method
Communication tests with enthusiasts focused on their comprehension. These tests took place in the main hall of IDE and in the living labs of the IDE faculty. The first test focused on communicating concepts and could be done without context. The second test was fully integrated and thus made use of the living environment. See the details in appendix G.
Setup similar to the usability testing, this time with digital interface and a disconnected prototype
Camera footage of the test in context
Participant in context, learning the dimmer through onboardin
Concluding
By using research through design, this project was steered in a controlled manner. The pivots in this project were planned and executed with the maximum amount of information available at the time.
Three pivots
The first pivot focused on the design scenarios that were developed through the context analysis. This resulted in successful solutions and thus the following phase pivoted to refine these design. The refinement resulted in a tangible solution that solidified a good experience for the non-enthusiast. After this, the enthusiast needed to be brought on board as well. The last pivot successfully designed a way to communicate usage and responsibilities to enthusiasts.
3 methods, 8 tests, 9 prototypes and 54 participants
These pivots made use of three different methods of validation. The methods recreated the design scenarios, isolated usability changes and tested enthusiast comprehension. In total 8 tests were executed resulting in 9 different prototypes, each prototype building on the knowledge of the previous tests and prototypes. The 8 tests had a total of 54 participants.
Research finds and a design
This method created many insights and many designs. The final results of this method are distilled research findings and a design with these findings properly implemented.