Designing Smart Home Appliances

Displaying Non-Urgent Everyday Information

Von der Fakultät für Informatik, Elektrotechnik und Informationstechnik

und dem Stuttgart Research Centre for Simulation Technology

der Universität Stuttgart zur Erlangung der Würde eines Doktors der

Naturwissenschaften (Dr. rer. nat.) genehmigte Abhandlung

Vorgelegt von

Alexandra Voit

aus Stuttgart

Hauptberichter: Prof. Dr. Niels Henze
Mitberichter: Prof. Dr. Jonna Häkkilä
Mitberichter: Prof. Dr. Paweł W. Woźniak

Tag der mündlichen Prüfung: 14.05.2021

Institut für Visualisierung und Interaktive Systeme

der Universität Stuttgart

2021





Zusammenfassung

Im Zeitalter des Smart Homes verändern sich die Wohnumgebungen erheblich.
Smart Home-Technologien bieten den Benutzern neue Möglichkeiten um in-
telligente Haushaltsgeräte zu steuern oder zu überwachen. In naher Zukunft
wird es notwendig sein, dass intelligente Haushaltsgeräte ihre Benutzer über
Alltagsinformationen zu Hause informieren, z. B. über den aktuellen Status des
Haushaltgerätes. In früheren Arbeiten wurde bereits die Benutzer-Akzeptanz von
Benachrichtigungen zu Alltagsinformationen im Smart Home untersucht. Bisher
wurde jedoch nur wenig untersucht, in welcher Form die Alltagsinformationen an
die Benutzer weitergeleitet werden sollen.

In dieser Arbeit wird untersucht, wie intelligente Haushaltsgeräte, welche
Alltagsinformationen an den Benutzer übermitteln konnen können, gestaltet wer-
den sollten. Diese Dissertation enthält die Ergebnisse zu neun durchgeführten
Benutzerstudien, die entweder den Nutzungskontext sowie die Benutzeranfor-
derungen für solche Haushaltsgeräte analysieren oder welche die entwickelten
Beispielanwendungen evaluieren. Hierbei konzentriert sich die Untersuchung
darauf, welche Modalitäten, Standorte und Informationsstrategien verwendet wer-
den sollten, um Alltagsinformationen in einem Smart-Home an die Benutzer zu
übermitteln. Darüber hinaus wird betrachtet, wie solche Haushaltsgeräte gestaltet
werden sollten, sodass sie zu den komplexen Routinen im Alltag der Benutzer
passen. Darüber hinaus wird untersucht, welche Evaluierungsmethoden für die

3



Bewertung von intelligenten Haushaltsgeräten geeignet sind, welche Alltagsinfor-
mationen darstellen können. Der Beitrag dieser Arbeit enthält Einblicke in Vor-
und Nachteile für verschiedene Bewertungsmethoden sowie Gestaltungsrichtli-
nien für die Entwicklung von intelligente Haushaltsgeräten, welche alltägliche
Informationen an den Benutzer übermitteln können.

4



Abstract

In the smart home era living environments are significantly changing. Smart
home technologies offer new opportunities for the users to control or monitor
their smart home appliances. In the near future, smart home appliances may
need to inform their users about everyday home details, such as their current
states. Previous work already investigated the users’ acceptance of smart home
notifications presenting everyday information. However, little research has been
done on how users can access the everyday home information.

This thesis examines how smart home appliances presenting everyday home
information should be designed. It reports about nine user studies investigating
either the context of use or the user requirements for smart home appliances
presenting everyday home information or evaluating the design solutions for the
investigated research probes. As a result of this, we focus on which modalities, lo-
cations and information strategies should be used to convey everyday information
in a smart home context. In addition, we study how smart home appliances cab be
designed to suit to the users’ complex daily routines. We further investigate which
evaluation methods are suitable for evaluating of smart home appliances present-
ing everyday home information. This thesis contributes insights into advantages
and disadvantages for various evaluation methods and design guidelines for the
development of smart home appliances that present everyday home information.

5





Acknowledgements

This work would have not been possible without the great support in the last years.
Therefore, thank you very much for all the great discussions that resulted in the
end in this thesis - even if I might have forgotten to mention you.

At first, I want to thank Niels Henze for supervising me as a PhD student
and supporting me during the whole time with his great advise and putting no
pressure on me to finally finish my PhD and not torturing me too often with
Helene Fischer’s music. Otherwise, I might have handed in sooner. :)

I would like to thank Paweł W. Woźniak for always being positive and
motivating and participating in the commitee and all the great discussions we had
in the last years. We should stay in touch and check in 2042 how smart homes
will really look like and proof if my vision was correct.

Furthermore, I want to thank Jonna Häkkilä and Thomas Ertl for being part
of the commity and Dominik Göddeke for being part in the mid-term evaluation’s
committee.

Thanks to Albrecht Schmidt for the great discussions and the useful feedback
during the years and a special thanks for always feeling responsible even if I
officially belonged to Niels’ group. A huge thanks for providing the funding that
enabled me to finish my investigations before moving on to industry.

7



Thanks to Stefan Schneegass for supervising me multiple times during my
studies - including the great Diploma thesis - and suggesting me for a PhD position
and; therefore; successfully intervening my decision to go to industry in 2015.

Great thanks to Dominik Weber for the great and amazing cooperations
and for always being reliable, motivating, and all the supportive chats, the great
discussions to investigate notifications from various angles, and his Photoshop
skills.

Thanks to Yomna Abdelrahman, Mariam Hassib, Passant El.Agroudy for
teaching me a little bit of Arabic - I promise I will never use at the airport or use
the swear words that Passant taught me and could get me killed if they are heard
by others on the streets. Shoukran!

Thanks to Nitesh "Tesh" Goyal, Christoph Anderson and David Dobbel-
stein for the great time in Singapore. it was a lot of fun and a pleasure to hang
out and explore Singapore with you all night long.

Also, I want to thank the other former members of the Stuttgart interaction lab /
hcilab group - the troop - who made my days in the lab bright and are now working
at different places within the world: Bastian "Basti" Pfleging, Celine Coutrix,
Francisco "Pepe" Kiss, Hyunyoung Kim, Huy Viet Le, Jakob Karolus Lars
"Immobilien" Lischke, Markus "Maku" Funk, Matthias Hoppe, Maura
Avila, Miriam Greis, Norman Pohl, Romina Poguntke, Rufat "Rufi" Rzayev
- thank you for being in the lab at lunch time, Sven Mayer, Thomas Kosch,
Thomas Kubitza, Tilman Dingler - I’m sorry for keeping your books at a safer
location, Tonja Machulla, Patrick Bader, Valentin "Vali" Schwind.

Thanks to the administrative staff from the University of Stuttgart who made
my life easier by organizing various things: Barbara Teutsch, Anja Mebus,
Murielle Naud-Barthelmeß, and Eugenia Komnik

Thanks to Andreas Bulling for still having a desk at the University (while
having contract with the University) after Andreas took over the HCI department
after Albrecht and Niels left.

Thanks to Michael Sedlmair for agreeing to being the examiner for a few
Bachelor students under my supervision after Niels and Albrecht both left the
University of Stuttgart.

8



Thanks to all the colleagues from other groups for the great collaborations.
It was a pleasure to be able to work with you: Katrin Wolf from Beuth Uni-
versity of Applied Sciences Berlin, Karola Marky and Alina Stöver from TU
Darmstadt, Jasmin Niess from University of Bremen, Elisabth "Lily" Stowell
from North-Western University, Kathrin Angerbauer from the University of
Stuttgart, Svenja Schröder from the University of Vienna, Kai Kunze from Keio
University, and Caroline Eckerth from University of Munich.

Further, I want to thank all colleagues from the DAAN project: Frederik
Wiehr, Sven Gehring and Antonio Krüger from the DFKI, Christoph Witte,
Daniel Kärcher and Steffen Süpple from Intuity Media Lab, Stefan Kohn
and Mirjiam Beljaars from Deutsche Telekom, Hauke Behrend and Catrin
Misselhorn from the University of Stuttgart, and Ebba Fransén Waldhör from
Berlin University of Arts.

Thanks to all the great students who I supervised within my time at the Univer-
sity of Stuttgart. It was (most of the time) a pleasure to be able to work with all of
you. Especially, I want to thank the students whose work contributed to this thesis:
Marie Olivia Salm, Henrike Weingärtner, Maike Ernst, Andres Michaela
Klapper, Manuel Müller, Amil Imeri, Anton Tsoulos, Daniel Koch, Kai
Chen, Marcus Rottschäfer, Robin Schweiker, Valentino Sabbatino, Annika
Eidner, Steven Söhnel, Nicole Krawietzek, Daniz Aliyev, Bernd Jung

A big thanks to all the participants in the conducted studies - also without you
this work would have not been possible!

Thanks to my computer science teacher U. Volck in high school who sup-
ported my interest for computer science including organizing a Mentor Program
in cooporation with IBM Deutschland.

Finally, thanks to my parents Annette and Hartmut M. Voit for support-
ing me throughout my life. Thank you both for giving me strength to chase my
dreams. Furthermore, I want to thank my sister Katharina Voit and my boyfriend
Franz G. Jahn for their love, support and the ability to distract or encourage me
whenever it was necessary.

Thank you very much!

9





Table of Contents

1 Introduction 17

1.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.2 Methodology and Evaluation . . . . . . . . . . . . . . . . . . . . . 20
1.3 Challenges and Research Contributions . . . . . . . . . . . . . . . 22
1.4 Research Context . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.5 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2 Background 31

2.1 Designing for the home . . . . . . . . . . . . . . . . . . . . . . . . 32
2.2 Smart home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.2.1 Smart home reminder and notification systems . . . . . . . . 34
2.2.2 Privacy in the smart home . . . . . . . . . . . . . . . . . . 35

2.3 Conversational Agents & Smart Speakers . . . . . . . . . . . . . . 38
2.3.1 Anthropomorphism of conversational agents & smart speakers 41
2.3.2 Abandonment and non-use of smart speakers . . . . . . . . . 41

2.4 Ambient information systems . . . . . . . . . . . . . . . . . . . . . 43
2.4.1 Taxonomies for ambient information systems . . . . . . . . . 43
2.4.2 Evaluation of ambient information systems . . . . . . . . . . 45

2.5 Mobile Notifications . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.5.1 Awareness and perception of notifications . . . . . . . . . . . 47

11



2.5.2 Negative effects of notifications . . . . . . . . . . . . . . . . 48
2.5.3 Effects of mobile unavailabilty . . . . . . . . . . . . . . . . . 49
2.5.4 Reducing negative effects caused by notifications . . . . . . . 50

2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3 Evaluation Methods for Smart Home Artifacts 53

3.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.1.1 Empirical Methods in human-computer interaction (HCI) . . . . 54
3.1.2 Comparison of Empirical Methods . . . . . . . . . . . . . . . 56
3.1.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

3.2 Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.2.1 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.2.2 Smart Artifacts . . . . . . . . . . . . . . . . . . . . . . . . 59
3.2.3 Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.2.4 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.2.5 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.2.6 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . 67

3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.3.1 Questionnaire Scores . . . . . . . . . . . . . . . . . . . . . 67
3.3.2 Item Reliability . . . . . . . . . . . . . . . . . . . . . . . . 70
3.3.3 Questionnaire Completion Time . . . . . . . . . . . . . . . . 71
3.3.4 Word Count Analyzes . . . . . . . . . . . . . . . . . . . . . 71
3.3.5 Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . 72

3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4 Exploration of Displaying Smart Home Notifications 81

4.1 Exploration of Modalities and Locations to Represent Home Information 82
4.1.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.1.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

4.2 Investigation of Locations for the Representation of Information in the

Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.2.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

12



5 Long-term Deployment for the Investigation of Notification Strategies

and Locations 99

5.1 Definition of Non-urgent Notifications . . . . . . . . . . . . . . . . . 100
5.2 Smart plant system . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.3 Focus Group Study . . . . . . . . . . . . . . . . . . . . . . . . . 102

5.3.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.4 System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

5.4.1 Notification Types . . . . . . . . . . . . . . . . . . . . . . . 108
5.4.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.4.3 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.4.4 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.4.5 Measurements . . . . . . . . . . . . . . . . . . . . . . . . 112

5.5 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.5.1 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.5.2 Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.5.3 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.5.4 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . 116

5.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.6.1 Quantitative Analysis . . . . . . . . . . . . . . . . . . . . . 118
5.6.2 Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . 121

5.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

6 Effects of Personal Content in Domestic Environments 131

6.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
6.2 Understanding the Calendar usage of Retirees . . . . . . . . . . . . 134

6.2.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
6.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

6.3 Caloo: Investigation of an Ambient and Pervasive Smart Wall Calendar

Supporting Event-Suggestions . . . . . . . . . . . . . . . . . . . . 138
6.3.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
6.3.2 Development of a Smart Wall Calendar Application . . . . . . 142
6.3.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
6.3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

13



6.4 Comparing novel Smart Home Appliances displaying calendar data with

established personal devices . . . . . . . . . . . . . . . . . . . . . 150
6.4.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
6.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

6.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

7 Understanding digital assistants in context 161

7.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
7.1.1 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 162
7.1.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . 163
7.1.3 Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . 164
7.1.4 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 166

7.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
7.2.1 Concerns and Nuisances . . . . . . . . . . . . . . . . . . . 166
7.2.2 Establishing Habits . . . . . . . . . . . . . . . . . . . . . . 169
7.2.3 Building Rapport . . . . . . . . . . . . . . . . . . . . . . . 170
7.2.4 Relationship Formation Over Time . . . . . . . . . . . . . . 172

7.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

8 Conclusion 181

8.1 Summary of Research Contributions . . . . . . . . . . . . . . . . . 182
8.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

Bibliography 191

List of Definitions 225

List of Figures 227

List of Tables 229

List of Acronyms 231

14



Please consider before reading:
To keep this thesis consistent, the scientific plural is used in this thesis.





1
Introduction

Through the smart home, living environments are significantly changing. Smart
home appliances will not only be connected, but they will also connect with
their users. Today, many users are using a large number of smart home appli-
ances, including smart speakers, security systems, smart heating, robotic vacuum
cleaners, or smart weather stations. In the near future, we assume that users will
regularly use a constantly increasing number of smart home appliances to monitor
or control them.

Smart home appliances have to provide consumers with a variety of informa-
tion about their homes regularly. A robotic vacuum cleaner, for example, must
inform users when its dust bag needs to be replaced, or a washing machine must
inform users when the laundry process is complete. Previous work investigated
the acceptance of home reminder systems that could use notifications to present
everyday information [114, 186, 187], but it was unclear how these everyday
home information should be communicated to users.

Mobile notifications are an established communication channel to present a
large amount of information to the users, including incoming messages, upcoming
appointments, or available updates [147, 161]. However, a body of work shows
that the number of mobile notifications that users receive daily overwhelms
them. Further, notifications are causing negative effects, such as distractions,

17



interruptions from current tasks, and even stress [11, 42, 88, 109]. These adverse
effects will be amplified if smart home appliances used mobile notifications on
users’ smartphones to display everyday home information to them.

It is necessary to consider the already overwhelming amount of notifications
users receive when designing smart home appliances that can alert them about
everyday home information. Current technologies enable many options to present
information in the home environment, e.g., by using ambient information systems.
However, smart home appliances must be designed for domestic use [52] and fit
within the home environment.

"I don’t think that simple home appliances – stoves, washing ma-
chines, audio and television sets – should look like Hollywood’s
idea of a spaceship control room. They already do, much to our
consternation."
(Don Norman [134])

Designers for smart home appliances must be aware of the complex routines
within the home [52] and consider the existing infrastructure [179]. Therefore, it
is essential to explore the design space, investigate and evaluate research probes
for different smart home appliances to be able to derive design guidelines for
the design of smart home appliances displaying everyday home information.
Therefore, the research probes should investigate different kinds of information
and different strategies for delivering everyday home information to the users,
such as whether a smart home appliance should notify its users proactively or
only upon when the users request for it.

1.1 Research Questions

This thesis explores how smart home appliances should be designed to present
everyday home information by investigating research probes for novel smart home
appliances. The exploration is based on six high-level research questions (RQs).
Table 1.1 lists the research questions (RQs) that are investigated in this thesis.

A thorough understanding of the methods for evaluating smart home appli-
ances appliances is an important basis for our exploration. Advances in technology

18 1 | Introduction



enable us to use augmented reality (AR) and virtual reality (VR) for rapid proto-
typing and collecting feedback for early prototypes. However, since it is currently
unclear whether the evaluation method influences the investigation results, it is
important to study which methods are suitable for evaluating early prototypes
of smart home appliances (RQ1). To be able to choose the right evaluation
method for an investigation, it is important to understand their advantages and
disadvantages (RQ2).

It is important to examine how smart home appliances can display home
information and how they can make users proactively aware of everyday home
information. First, it is critical to investigate which modalities are suitable for
making users proactively aware of home information when designing smart home
appliances displaying home information (RQ3). Also, it is important to investigate
which locations in the home are suitable to display everyday information (RQ4).
Further, it needs to be examined whether home information should be persistently
displayed, though the users can perceive continuous changes in the information or
whether the users should be only made aware of the information based on specific
events such as actions that need to be accomplished (RQ5).

It is also essential to investigate smart home appliances where the user initiates
the interaction with the smart home appliance. In this case, appliances present the
information unobtrusively or display the information only to the users based on a
user’s request (i.e., user-poll mechanism). Hereby, it is important to study how
such smart home appliances can be designed to fit into the users’ routines (RQ6).

1.1 | Research Questions 19



Research Question No. Chapter

Understanding evaluation methods regarding smart home appliances

What are the suitable evaluation methods to
study smart home appliances informing the
users about everyday information?

(RQ1) Chapter 3

What are the advantages and disadvantages
of different methods for the evaluation of
smart home appliances?

(RQ2) Chapter 3

Making users proactively aware of everyday home information

Which modalities are suitable to inform
users about everyday information in the era
of the smart home?

(RQ3) Chapter 4, Chapter 6

Which locations are suitable to display ev-
eryday information in a smart home?

(RQ4) Chapter 4, Chapter 5

Should smart home information be persis-
tently displayed to the users or be made
aware of specific events?

(RQ5) Chapter 5

User-initiated interaction with smart home appliances presenting
everyday information

How should smart home appliances be de-
signed to fit into the users’ routines?

(RQ6) Chapter 6, Chapter 7

Table 1.1: Overview about the RQs that are investigated in this thesis.

1.2 Methodology and Evaluation

"Design presents a fascinating interplay of technology and psychol-
ogy, that the designers must understand both."
(Don Norman [134], Chapter 1, p. 7)

Designing, developing, and evaluating novel applications is an essential part of
human-computer interaction research. In 1985, Gould and Lewis defined three key
principles for the development of usable and easy to use computer systems [72]:
(1) focusing early on the users by studying the characteristic of the users and
understanding the context of use, i.e., the user types, their tasks, the resources and

20 1 | Introduction



Figure 1.1: The human-centered design process consists of the following phases:

specifying the context of use, specifying the user requirements producing design

solutions as well as evaluating the design solution against the user requirements until

all user requirements are met by the developed design solution

the environment (2) using empirical measurements observe the performance and
reactions of intended users and (3) applying an iterative design by repeating the
steps of design, test, and measure as often as necessary.

The research conducted in this thesis is inspired by the human-centered-design
process (IS0 9241-210) [63]. To investigate the research questions different
research probes will be developed and evaluated. The human-centered design
process (see Figure 1.1) consists of the following iterative four phases [63, 182]:

Understand and specify the context of use: During this step, the context of use
is studied by taking into account the environment, the tasks, and the intended

1.2 | Methodology and Evaluation 21



users. The context of use can be investigated using different techniques, e.g., by
conducting observations, interviews, focus groups, or online surveys. The context
of use is specified based on the observations.

Specifying the user requirements: Afterwards, the intended system’s user re-
quirements are specified.

Produce design solutions to meet the user requirements: The design solutions
will be developed to meet the user requirements. These design solutions can be
prototypes with different fidelity, e.g., with a low fidelity such as paper prototypes.

Evaluate the design against the user requirements: The developed design so-
lution will be evaluated, e.g., by conducting studies or applying heuristic in-
spections [132]. If the evaluation reveals that the prototype fulfills the user
requirements specified for the developed system, the iterative process ends. If the
prototype did not meet the user requirements, the steps are repeated until the user
requirements are fulfilled. With this, it is important to note that not all phases
from the human-centered design process need to be repeated, e.g., by revising the
produced design solutions.

1.3 Challenges and Research Contributions

By investigating the research questions (RQs) listed in Table 1.1, we make the
following contributions: In Chapter 3, we compare five evaluation methods for
studying smart home appliances. Here, we contribute that empirical methods can
significantly affect the outcome of user studies indicating that results from studies
using different empirical methods might not be comparable. In Chapter 4, we
conduct an exploration of the design space of smart home notifications. Here, we
contribute an understanding of the relation between the information’s urgency and
the used modality and location for the representation in the home. In Chapter 5,
we study a smart plant system as a research probe informing the users proactively
about the plant’s current state. We contribute a systematical analysis of different
strategies to present non-urgent smart home notifications.

22 1 | Introduction



In Chapter 6, we investigate calendar applications as research probes. We
contribute an understanding of how smart home appliances giving information
that users can manually check should be designed to fit the users’ routines. In
Chapter 7, we observe how novel users integrate smart speakers by conducting
a four-week in-situ study. We contribute implications to improve the design of
future smart speakers. An overview of all research probes that were developed
and investigated in the context of this thesis is displayed in Table 1.2.

1.4 Research Context

The research that led to this thesis was conducted from 2015 to 2019 in the
Institute of Visualization and Interactive Systems at the University of Stuttgart
in the Department of Socio-cognitive systems under the supervision of Prof. Dr.
Niels Henze. Furthermore, this research was conducted within the context of
the Graduate School Simulation Technology of the Stuttgart Center. This thesis
was subjected to a mid-term evaluation by Prof. Dr. Niels Henze and Prof. Dr.
Dominik Gödekke regarding the rules of the Cluster of Excellence in Simulation
Technology (SimTech) Graduate School.

Funding The main part of this thesis was conducted with the scope of the
project “Designing adaptive and ambient notifications (DAAN)” at the University
of Stuttgart. The project DAAN was funded by the Federal Ministry for Education
and Research in Germany (BMBF). After the BMBF project ended, I was founded
by base funding until the end of 2018. Afterward, I was founded by remaining
funds from the former Department for Human-Computer Interaction of Prof. Dr.
Albrecht Schmidt.

1.4 | Research Context 23



Picture Research probe Description Chapter

Smart artifacts The cup saucer displays
the coffee’s temperature.
The stand indicates
the filling level for the
respective mill. The
speaker displays the
volume of the played
music. The plant pot
displays if the plants
needs water.

Chapter 3

Plant system The smart plant system
displays the plant’s wa-
ter level either directly on
the plant pot or on the
user’s smartphone.

Chapter 5

Wall calendar The wall calendar dis-
plays the user’s schedule
and event suggestions fit-
ting the user’s interests.

Chapter 6

Calendar data
representations

Calendar representations
using established dis-
plays and novel displays,
such as smart lights or
e-paper displays.

Chapter 6

Table 1.2: Overview about the research probes that were developed in this thesis.

24 1 | Introduction



Previous work The research presented in this thesis is based on the following
publications that were published at international scientific conferences and work-
shops: [192–194, 196–199, 201, 203]
Please consider that the scientific plural is used in this thesis.

The following collaborative efforts contributed to the described research probes
and user studies in this thesis:

Chapter 3 - Evaluation Methods for Smart Home Artifacts: The author sug-
gested the idea of the described study; the final study design was developed
in discussions between the author, Sven Mayer, Valentin Schwind, and
Niels Henze. The implementation of the apparatus was distributed between
the author (i.e., AR prototypes), Sven Mayer (i.e., VR and physical proto-
types), and Valentin Schwind (i.e., modeling the physical objects for VR
and developing the used online survey). The author executed the study. The
statistical analysis was conducted by Valentin Schwind, while the author
executed the analysis of the qualitative data with the support of Valentin
Schwind. The resulting publication was written by the author with the input
from Sven Mayer, Valentin Schwind, and Niels Henze [201].

Chapter 4 - Exploration of Displaying Smart Home Notifications: This
chapter reports two user studies. The first study investigating modali-
ties and locations for displaying information in a smart home is based on
the student project of Nicole Krawietzek, Daniz Aliyev, and Bernd Jung.
The project was supervised by the author, Dominik Weber, and Stefan
Schneegass. The students conducted the reported focus groups. Stefan
Schneegass wrote the resulting publication with the author’s and Dominik
Weber’s support [192].
The author stemmed the idea, concept, study design, and the execution of
the second user study. Valentin Schwind provided the investigated pictures
presenting the notification types. Tonja Machulla analyzed the collected
data. The resulting paper was mainly written by the author and Stefan
Schneegass with the support of Tonja Machulla, Dominik Weber, Valentin
Schwind, and Niels Henze [198].

1.4 | Research Context 25



Chapter 5 - Long-term Deployment for the Investigation of Notification
Strategies and Locations: The described system was developed in Marie
Olivia Salm’s Bachelor thesis. The author primarily supervised the Bach-
elor thesis. The student did the focus groups, implementation, and a part
of the evaluation. The author also conducted a part of the evaluation. The
author analyzed the qualitative data with the support of Dominik Weber
and Paweł W. Woźniak, while the author conducted the statistical analysis.
The author primarily wrote the resulting publication with the input of Do-
minik Weber, Yomna Abdelrahman, Paweł W. Woźniak, Stefan Schneegass,
Katrin Wolf, and Niels Henze [197].

Chapter 6 - Effects of Personal Content in Domestic Environments: This
chapter describes a total of four user studies. Elizabeth Stowell designed
the first described user study reporting the calendar usage of retirees with
the support of the author. The author analyzed the survey with the support
of Elizabeth Stowell and Dominik Weber. The idea and concept of the
second reported user study was stemmed from the author. The apparatus
was developed by Dominik Weber and Steffen Süpple from Intuity Media
Lab named the prototype. The author and Dominik Weber conducted the
lab study. The author executed the analysis of the collected data. The paper
reporting the two user studies was written by the author with the input of
Dominik Weber, Elizabeth Stowell, and Niels Henze [193].
The third user study that investigated the implementation and evaluation of
the Caloo system was developed in Manuel Müller’s Master thesis project.
The student did the implementation of the system and the described user
study. The author primarily initiated and supervised the project with Rufat
Rzayev’s support. The author wrote the published paper with the input and
support of Rufat Rzayev, Dominik Weber, and Niels Henze [199] .
The last described user study in this chapter was part of the student software
project from Amil Imeri, Anton Tsoulos, Daniel Koch, Kai Chen, Marcus
Rottschäfer, Robin Schweiker, Valentino Sabbatino, Annika Eidner, Steven
Söhnel. The student project was envisioned in discussions between the
author and Dominik Weber. Further, the student project was supervised by
the author and Dominik Weber. The students implemented the apparatus

26 1 | Introduction



and conducted the user study. The author conducted the data analysis. The
author wrote the resulting publication with the input from Dominik Weber
and Niels Henze [196].

Chapter 7 - Understanding digital assistants in context: The idea and con-
cept of the reported user study was stemmed by the author with the support
of Paweł W. Woźniak. The study was conducted as a part of the student
project of Henrike Weingärtner, Maike Ernst, and Andres Michaela Klap-
per. The data analysis was executed by the author, Jasmin Niess, Paweł W.
Woźniak and the support of Caroline Eckerth. The resulting publication
was written by the author, Paweł W. Woźniak, and Jasmin Niess [203].

The papers presented in this thesis are motivated by a cooperation with
Dominik Weber and his research about managing notifications on established
personal devices: [7, 191, 208–210, 212, 213, 215–217]

Within the context of the research presented in this thesis, multiple workshops
focusing on attention management were organized: [57, 140, 185, 202, 214, 218]

Cooperation within the Institute of Visualization and Interactive Systems In
addition to the publications related to this thesis topic co-operations within other
researchers in the institute has resulted in the following publications: [8, 10,
96, 193, 200, 211, 219]. A cooperation regarding smart fabrics with Stefan
Schneegass resulted in the following publications: [167, 189, 190]
Especially successful was a cooperation with Katrin Angerbauer, Sven Mayer
and Michael Sedlmaier which lead to a publication at IEEE VIS [219]. Another
cooperation with Patrick Bader, Huy Viet Le, Niels Henze and Albrecht Schmidt
led to an article published in ACM Transactions on Computer-Human Interaction
(TOCHI) [10].

External Collaborations Other collaborations with external researchers or other
companies lead to the following contributions:

Miriam Beljaars and Stefan Kohn from Deutsche Telekom AG [194], Frederik
Wiehr, Sven Gehring and Antonio Krüger from the German Research Center
for Artificial Intelligence (DFKI) [220], Christoph Witte and Daniel Kärcher
from Intuity Media Lab GmbH [195, 220], Karola Marky, Stöver Alina and Max

1.4 | Research Context 27



Mühlhäuser from the Technical University of Darmstadt and Kai Kunze from
Keio University and Svenja Schröder from University of Vienna [110], Jasmin
Niess from University of Bremen and Caroline Eckerth from University of Mu-
nich [203]
In cooperation the following workshops at international conferences were orga-
nized: [47, 168]

1.5 Thesis Outline

This thesis consists of eight chapters, followed by the bibliography and enumerat-
ing lists for figures and tables. The thesis is structured as follows:

Chapter 1 - Introduction: This chapter contains the description and motivation
of this thesis. Further, it presents an overview of the research questions
(RQs) and contributions. Lastly, it contains this outline.

Chapter 2 - Background: This chapter describes the former work that is
relevant to this thesis. It contains the former work about designing for
the home, about the smart home, and conversational agents. Finally, it
describes previous work about ambient information systems and mobile
notifications.

Chapter 3 - Evaluation Methods for Smart Home Artifacts: This chapter
studies the evaluation method online survey, studies using VR and AR,
as well as established lab and in-situ studies for the evaluation of early
prototypes of smart home appliances. We show that the method used
for evaluation can significantly affect the outcome of the investigation,
e.g., studies using VR or AR can get affected by novelty effects. This
investigation addresses RQ1 and RQ2.

Chapter 4 - Exploration of Displaying Smart Home Notifications: This
chapter explores how and where everyday information could be displayed
in the home environment by conducting focus groups and an online survey.
This investigation addresses RQ3, RQ4, and RQ5.

Chapter 5 - Long-term Deployment for the Investigation of Notification
Strategies and Locations: In this chapter a smart plant system informing

28 1 | Introduction



the users about the plant’s water level is developed as a research probe.
A long-term in-situ study is conducted that investigates different strate-
gies to display notifications as well as different locations to display such
notifications. This study addresses RQ4 and RQ5.

Chapter 6 - Effects of Personal Content in Domestic Environments: This
chapter investigates the representation of calendar data in the home. Using
different studies, we observe how our research probes displaying calendar
information fit the users’ routines. This investigation addresses RQ3 and
RQ6.

Chapter 7 - Understanding digital assistants in context: This chapter ob-
serves how smart speakers are integrated into the users’ routines when they
are newly introduced in a home environment. Furthermore, we investi-
gate how users experience smart speakers in their daily lives. This study
addresses RQ6.

Chapter 8 - Conclusion: This chapter summarizes and discusses the results
presented in the former chapters. In addition, we present directions for
future research to build upon this thesis.

1.5 | Thesis Outline 29





2
Background

This thesis investigates how smart home appliances should convey everyday home
information to their users. Consequently, the research reported in this thesis
builds upon previous work from the following research strands: designing for the
home, smart home, conversational agents & smart speakers, ambient information
systems and mobile notifications.

This chapter is partly based on the following publications:

A. Voit, D. Weber, Y. Abdelrahman, M. Salm, P. W. Wozniak, K. Wolf, S. Schneegass, and
N. Henze. “Exploring Non-Urgent Smart Home Notifications Using a Smart Plant System.”
In: 19th International Conference on Mobile and Ubiquitous Multimedia. MUM 2020. Essen,
Germany: Association for Computing Machinery, 2020, 47–58. ISBN: 9781450388702.
DOI: 10.1145/3428361.3428466

A. Voit, J. Niess, C. Eckerth, M. Ernst, H. Weingärtner, and P. W. Wozniak. “‘It’s Not a
Romantic Relationship’: Stories of Adoption and Abandonment of Smart Speakers at
Home.” In: 19th International Conference on Mobile and Ubiquitous Multimedia. MUM
2020. Essen, Germany: Association for Computing Machinery, 2020, 71–82. ISBN:
9781450388702. DOI: 10.1145/3428361.3428469

31

https://doi.org/10.1145/3428361.3428466
https://doi.org/10.1145/3428361.3428469


2.1 Designing for the home

Designing and exploring interactive systems for the home has a long tradition
in HCI research. Almost two decades ago, Edwards and Grinter [52] identified
seven challenges in the context of ubiquitous computing for the home. For
instance, one challenge the authors discussed was ‘Designing for Domestic Use’.
They emphasized the need for designers to build an in-depth understanding of
the home’s complex routines and how these might lead to the adoption or the
abandonment of new technologies. We aim to shed more light on these processes
and their consequences. Nylander et al. investigated which kind of computing
devices users generally prefer using [136]. They found that users prefer using
mobile phones at home since they considered performing tasks on phones due to
their availability as quicker and easier. Grinter and Edwards further investigated
how households make their home network function and found that they need
to coordinate the usage of different home appliances and the effort to configure
and manage the appliances [74]. Further, Tolmie et al. found that designers for
networking technologies in domestic environments need to consider the existing
infrastructure in the home and how future technologies can be integrated into
existing routines [179].

Former work also investigated how technology can be introduced into do-
mestic environments. For example, Crabtree et al. identified places in the home
(i.e., ecological habitats, action, and coordinate displays), such as tables or no-
tice boards that habitually draw the users’ attention [38]. They identified those
places as prime sites for ubiquitous computing applications that support everyday
activities and focus on sharing communication media between the residents in
a home context. In another vein, Crabtree and Tolmie [37] showcased in their
observational study that the assemblage and the arrangement of things are con-
nected to everyday routines. Earlier, Odom et al. [137] investigated how the
arrangement and presence of future technologies in teenage bedrooms might
affect self-exploration and identity construction processes.

We learned in this section that it is important to consider the specific con-
straints within the users’ homes when designing smart home appliances for
domestic use. Therefore, it is important for this thesis that we understand the

32 2 | Background



context of use and the users’ regular routines before designing research probes
presenting everyday home information. Further, we need to investigate how the
design of smart home appliances presenting everyday home information can fit
the users’ routines (RQ6). Here, we should investigate the usage of mobile phones
as their usage is experienced as convenient [136].

2.2 Smart home

Although smart home technology is around for multiple years, many users are still
not using them in their homes. Previous work observed the integration of smart
home technology [12, 69, 77, 222]. Balta Ozkan et al. investigated possible social
barriers hindering users adopting smart home technologies [12]. They found that
reasons for the lacking consumer acceptance include concerns regarding a loss of
control and apathy, privacy and data security, the reliablity of the devices and the
fact that the new smart home technology might be incompatible with other - espe-
cially older - appliances. Hargreaves et al. investigated the domestication of smart
home technologies in a long-term study [77]. They found that these technologies
can be disruptive, that the need for adoption and familiarization can limit their
use of the technology and that learning the functionalities of the technology is a
time-consuming task. Geeng and Roesner observed how smart home technologies
affect the existing relationships within a home and found smart home technologies
change the dynamics and lead to power structures [69]. Williams et al. found that
the upcycling of existing objects by light-weight modifications within a home
minimizes the risk of destabilizing domestic relationships and values [222].

Visualizing the smart home appliances’ states in a calendar form enables the
users to understand complex routines and increases their trust in the smart home
technology [118, 119]. Castelli et al. investigated a configurable dashboard
display that shows real-time information of the smart home on a room level
basis, integrates weather forecasts, and accesses the smart home’s configuration
settings [26]. They found that offering pre-defined visualizations is essential.
However, smart home appliances should offer opportunities to map data of the

2.2 | Smart home 33



smart home appliance easily to different kinds of smart home visualizations, such
as charts. However, the user needs to be able to adapt the visualization according
to their needs [26].

Technology advancements have enabled the ue of smart homes as well as
the support of new scenarios. Knierim et al. explored the design space of using
AR in domestic environments [94]. Although their participants saw a potential
for integrating AR into the homes, the authors concluded that new privacy and
transparency rules are needed.

2.2.1 Smart home reminder and notification systems

McGee-Lennon et al. showed that people from all age groups likely forget tasks
in and around their homes [114]. Specifically, they found that middle-aged and
younger people tend to forget more diverse tasks than older adults. Thus, there is a
need for home reminder systems supporting users of all age groups. It was found
in the former work that many people use paper-based reminders (e.g., notes),
people-based reminders (i.e., they ask someone to remind them) or physical
reminders (e.g., by placing objects close to the entrance door so as not to forget
to take them when leaving their homes) [91, 114]. Further, McGee-Lennon et al.
reported that people also integrate tasks into their routines or use technological
and specialized reminders in their homes to be aware of upcoming tasks or their
schedules [114].

An important factor for home reminder systems is their acceptance. If a
reminder system is not accepted, users might turn it off or ignore displayed
reminders [114, 187]. Previous work investigated factors that support the accep-
tance of technological reminder systems in domestic environments [114, 186,
187]. McGee-Lennon showed that reminder systems are more accepted when they
use metaphors or reminding strategies users are already used to [114]. Further,
the acceptance of displayed notifications in domestic environments depends on
the urgency of the notification [186, 187]. High-urgent notifications are more
accepted than non-urgent ones [186, 187]; medium-urgent notifications were
accepted when they were unobtrusively presented to the users [186]. In contrast,
low-urgent notifications were not accepted by their participants [186]. Their
results showed that low-urgent notifications should be delayed until the urgency

34 2 | Background



of the notification increases. If the urgency of a low-urgent notification does
not increase, it should not be displayed to the user. Furthermore, in contrast to
a notification’s urgency, the user’s current primary task does not influence the
acceptance of notifications in the users’ home environments [186].

Previous work investigated how different modalities to display notifications
in home environments affect a primary task [206, 207]. Warnock et al. found that
the modality affects the time required to perceive a notification, but has no effect
on disrupting a primary task [207]. However, Warnock et al. showed that the
modality does not influence primary task’s performance (i.e., error rate) [206].

A body of work investigated the design of future notifications systems [43,
192, 198]. Czerwinksi et al. found that devices in smart environments compete
for users’ attention [43]. Hence, there is a need to design notification systems
informing users by displaying information subtly. Bourgeois et al. found that
delayed and real-time feedback are not appropriate tools to support demand
shifting behavior; instead, proactive suggestions and contextual control support
users in organizing their daily lives by micro-planning and micro-scheduling
household activities [21]. Further, notification systems in the home should support
natural and transparent interactions [91].

Other work investigated home reminder systems that support their users in
certain routines (e.g., when leaving home) [91, 170]. Kim et al. developed a
home reminder system with a display close to the entrance door that reminds
the users of things they have to take with them when leaving their homes [91].
They reported that users do not want to interact explicitly with a home reminder
system. In contrast, users prefer natural and transparent interactions. Seiderer et
al. developed a system that uses ambient lighting to display if all critical doors
and windows are closed (e.g., to prevent burglaries when leaving home or going
to sleep) [170]. They found that their system improved the certainty of the state
of critical doors and windows.

2.2.2 Privacy in the smart home

Since smart home technologies can access or collect sensitive data of their users,
the users’ privacy must be considered during the development of smart home
appliances [224]. However, former work observed that their participants could

2.2 | Smart home 35



not name privacy consequences beyond general privacy issues such as data col-
lection [71, 89]. In addition, previous work showed that there is a trade-off
between the functionality a device offers or the users’ convenience and privacy
and security [225, 226]. An example of this trade-off is the opportunity to interact
remotely with a smart home appliance vs. storing the data in the cloud [226].

Other work investigated whether users trust companies or manufacturers of
devices with protecting their privacy [156, 177, 225]. Rodden et al. observed
that users in the UK are not trusting the energy providers in monitoring energy
data since they were concerned that the companies would use the data for their
own advantages [156]. In contrast, other study has revealed that users trust
device manufacturers without verifying if their trust is justified [225] or even
when they are aware of the devices’ privacy and security issues, such as of smart
speakers [177]. Reasons for trusting companies, e.g., manufacturers of a device
are that companies that cannot afford misusing the collected data because of
possible results of data misuse such as losing the reputation.

In addition, other research investigated the data collection of smart home
technologies regarding privacy [54, 100]. For example, Emami-Naeini found that
users prefer sharing data collected in public spaces than in private environments
such as homes [54]. Further, users are willing to share also sensitive data with
third parties such as manufacturers of smart home appliances as long as the
collected data is sufficiently abstracted and anonymized [100] or beneficial [54].
In addition, former work reported no privacy issues when environmental data is
collected [54]. However, biometric data should not be collected in any case [54].

Further, former work found that most users do not consider privacy before
purchasing or installing smart home appliances [53, 110]. However, these users
get concerned about privacy issues afterward [53], e.g., they were concerned
about sharing their data with potential bystanders in their home [110].

Current work also investigated how smart home technologies can affect the
inhabitants’ privacy regarding present bystanders in the home environment, such
as other inhabitants or visitors [110, 181]. Ur et al. found that teenagers were
concerned that their parents could monitor them closely if smart locks or entry-
way cameras would be installed in their various homes [181]. This means that
in addition to the trade-off between functionality and privacy regarding third

36 2 | Background



parties [226], the social dynamics within the home need to be considered before
purchasing or installing smart home appliances. In addition, Marky et al. in-
vestigated what kinds of information can be displayed within the home without
causing privacy issues by the presence of other bystanders such as other inhab-
itants or visitors who could also access the information [110]. They found that
the majority of their participants had no concerns about sharing their household
information with others; in addition, some participants stated that it would be even
beneficial if household data would be shared with other persons close to them.
The willingness to show privacy-sensitive information in the home is affected
by the location in the home where the information would be displayed [10, 110].
However, privacy options such as a visitor mode can make it possible to display
sensitive information such as calendar data in the home environment in places
that others can access [110].

In this section, we learned that visualizing smart home data is important to
increase users’ trust in the used smart home technologies [118, 119]. Another
important factor is that smart home technologies should offer opportunities for
their users to adapt the visualizations according to their needs [26]. However,
when designing smart home appliances, we need to consider that the integration
of smart home technologies can affect the social dynamics within the home [69,
181]. Williams et al. suggested enhancing already existing objects in the domestic
environments to reduce the risk of affecting the social dynamics within the home
[222]. Furthermore, we found that novel technologies such as AR can be used
to prototype and evaluate smart home appliances. However, we do not know
whether using novel technologies could affect the outcome of the conducted
evaluation. Therefore, we need to investigate which evaluation methods are
suitable for evaluating prototypes of smart home appliances that display everyday
information (RQ1). We must understand the advantages and disadvantages of
different evaluation methods to choose the right evaluation method for the specific
purposes of a study (RQ2).

Furthermore, we found that smart home appliances presenting everyday home
information can support users from all age groups to remind them about upcoming
home tasks. Former work already reported what kind of information the users
would like to receive based on investigating the acceptance of smart home appli-

2.2 | Smart home 37



ances for different kinds of information with different urgency levels. However, it
remains unclear how these kinds of home information should be conveyed to the
users. Therefore, it is important to study which modalities (RQ3), and locations
(RQ4) are suitable to inform the users about these kinds of information as well as
when the home information should be displayed (RQ5).

Finally, we learned that privacy issues could impact the acceptance and
usage of smart home appliances. Therefore, it is important to consider privacy
options when designing smart home appliances - especially when personal data
is displayed. In this case, a visitor mode could be integrated, or the users could
configure the smart home appliance’s output accordingly to its location in the
home. We also learned that the participants were not concerned when household
data was shared with other persons. As a result, research probes presenting
household information should not cause privacy issues.

2.3 Conversational Agents & Smart Speakers

Smart speakers feature a conversational agent, a new conversation partner de-
signed for regular ‘communication’ with the user. Since smart speakers are
becoming popular everyday devices in the users’ homes1, the functionality of
smart speakers could be extended to inform their users’ proactively everyday
home information to the users.

Research that addresses smart speakers is inherently related to understanding
conversational agents’ integrated conversational agents on mobile phones or
laptops. In contrast to other conversational agents, smart speakers are located at
a static position in the users’ homes [169]. Smart speakers have an integrated
conversational agent that can be activated through an activation word. For instance,
Amazon Echo, the smart speaker from Amazon, connects to the voice-controlled
conversational agent Alexa.

Former work that studied conversational agents revealed that most users could
not accurately judge the system capacities of conversational agents. Users without

1https://techcrunch.com/2020/02/17/
smart-speaker-sales-reached-new-record-of-146-9m-in-2019-up-70-from-2018/

38 2 | Background

https://techcrunch.com/2020/02/17/smart-speaker-sales-reached-new-record-of-146-9m-in-2019-up-70-from-2018/
https://techcrunch.com/2020/02/17/smart-speaker-sales-reached-new-record-of-146-9m-in-2019-up-70-from-2018/


knowledge of computer science were mostly missing an exact mental model of
how conversational agents work [105]. Unfortunately, that resulted in a tendency
to anthropomorphize the conversational agent and setting unrealistic expectations.

A body of work investigated how users interact with conversational agents [29,
35, 36, 51, 120, 149]. Clark et al. investigated differences between human-to-
human and human-to-agent conversations. They found that participants used
similar interlocutor characteristics in communication with strangers or casual
acquaintances and communication with conversational agents [29]. However,
their participants questioned the need for bonding and developing a relationship
with conversational agents. Porcheron et al. in their work showed that users
predominantly react to conversational agents’ failures by repeating the original
query or reformulating it [149].

Other work identified barriers that act as barriers for users of conversational
agents [35]. Users were often frustrated by the need to combine touch and speech
interaction to interact with conversational agents, e.g., selecting a contact to call
or unlocking the phone before a query can be entered [35]. Further, users prefer
to enter non-private data to conversational agents [51, 120] and use conversational
agents in safe or domestic environments [120]. Reported reasons for avoiding
speech interaction in public were mainly privacy concerns [120], embarrassment
in front of strangers [36, 120], and cultural factors [36].

Several recent works studied different aspects of how users interact with smart
speakers [14, 16, 150]. Bentley et al. [16] investigated habits in smart speaker
use through analyzing an extensive database of voice history logs from Google
Home. They found that playing music was by far the most used action on the
smart speaker. As the users owned the device longer, music usage was still high
but declined. On the other hand, users increasingly used more automation, sug-
gesting that the smart speaker was further integrated into the home environment.
Porcheron et al. [150] analyzed audio data from a month-long Amazon Echo
usage period to study the intricacies of the dialogues between users and smart
speakers. They found that the ‘atomic’ way one communicated with a smart
speaker bore little resemblance to real conversations. Beneteau et al. analyzed
the situation in which the communication between family members and the smart
speaker was breaking down and found that in these cases, family members often

2.3 | Conversational Agents & Smart Speakers 39



collaborate to repair the communication with the smart speaker by discoursing
scaffolding and varying the speech, e.g., the pronunciation, the language, such as
the used wording [14]. Sciuto et al. studied usage logs and conducted interviews
with users of Amazon Alexa [169]. They found that users of smart speakers
explored the functionality of smart speakers within the first few days. Afterward,
their participants used the smart speaker constantly week-over-week for the first
year. However, the number of daily requests varied between households.

Other work studied how users experience smart speakers [101, 154] or in-
vestigated privacy perceptions [106, 120]. Lau et al. conducted diary studies
and interviews with smart speaker users and interviews with non-users of smart
speakers about their experience while focusing on privacy perceptions [101].
They found that non-users do not see the utility in smart speakers, while users of
smart speakers are aware that they trade in their privacy for convenience. Users,
as well as non-users of smart speakers, are distrusting the speaker companies.
Manikonda et al. showed that smart speakers’ users prefer to use them in their
daily lives, although they are concerned about privacy, e.g., about being hacked
or about the data collection and data storage [106]. Furthermore, they showed
that even some tech-savvy users were not aware that smart speakers are always
listening. Once their users were made aware of this fact, the privacy concerns
increased significantly. Moorthy and Vu analyzed privacy and security issues
that are caused by conversational agents [120]. They showed by investigating
different possible attacks that the users’ interaction with the conversational agents
is the weakest link. Here, one of the reasons is the used (predefined) wake word
of the conversational agents that are easy to guess by others and can also be trig-
gered by external sources such as advertisements shown on the TV. Pradhan et al.
analyzed Amazon reviews and interviews to study how smart speakers supported
accessibility [154]. They found that smart speakers at home were perceived as
particularly beneficial by participants with a vision impairment who actively used
the devices to facilitate many daily actions.

40 2 | Background



2.3.1 Anthropomorphism of conversational agents & smart
speakers

There is an increasing interest in the HCI community to explore the intricacies
of designing future digital assistants using conversational agents [31]. However,
it still remains a challenge to design technologies that can become meaningful
digital assistants [133], i.e., technologies that carry personal and social meaning.
Lopatovska and Williams [104] reported that a significant share of Amazon
Echo’s users expected the device to exhibit social behavior. Preliminary results
by Purington et al. suggest that users tend to ascribe human qualities to smart
speakers despite the ‘non-human’ conversation style [155] . They analyzed
user reviews of the Amazon Echo and found that user satisfaction seems to be
connected to the technology’s more personification.

The class of behaviors mentioned above can be called anthropomorphism and
defined as the attribution of human-like characteristics, motivations, emotions,
or intentions to non-human agents, such as animals or objects [55]. Epley et
al. state that anthropomorphizing serves three purposes, namely: (1) making
sense of situations, (2) reducing uncertainty in specific situations, and (3) estab-
lishing social connections [55]. Early research in HCI found that humans react
towards technologies in social ways [125, 126]. However, Nass and colleagues
take another, contrasting stance compared to Epley et al. and state that these
social reactions are triggered by social cues [125]. Kuzminykh observed that
smart speakers’ anthropomorphization is related to their implemented behavior
regarding the categories approachability, sentiment, professionalism, intelligence,
and individuality, e.g., Alexa is perceived as genuine and caring while Siri is
perceived as cunning and disingenuous [99]. Further, Gao et al. found that an-
thropomorphizing smart speakers also show more positive emotions than users
who treated the smart speaker like an electronic device [65].

2.3.2 Abandonment and non-use of smart speakers

Some studies of smart speakers reported that participants reduced their inter-
actions over time [169], whereas other results indicate that there is no decline
regarding the usage [16]. However, to date, to the best of our knowledge, only
two studies [28, 67] have focused explicitly on the abandonment of a class of

2.3 | Conversational Agents & Smart Speakers 41



devices that included smart speakers. Cho et al. investigated smart speakers’
abandonment in a long-term diary study with first-time users of smart speak-
ers [28]. They found that the reasons for the abandonment of smart speakers
after a few weeks of usage are based on a disappointing exploration leading to
minimal usage or abandonment. Garg and Kim [67] conducted an exploratory
study to build an understanding of the usage of the Internet of Things (e.g., voice
assistants, smartwatches, smart locks). Their preliminary results showed that
participants mainly stopped using devices due to demotivating interactions (e.g.,
distracting notifications, notifications of failure to achieve a goal). Further, they
found that participants stopped using the device when it was too complicated
to use or provided unnecessary, confusing information. Only a few participants
mentioned privacy concerns as a determining factor regarding continued usage.
One of the relevant factors for continuing usage was autonomy in daily activities.

Other work investigated the reasons for the non-usage of devices [163, 164].
Satchell et al. found that the technology’s adoption could be lagging according to
active resistance by the users, disenchantment, disenfranchisement, displacement,
and disinterest [164]. Sambasivan et al. identified the reasons for avoidance
by the user (e.g., by turning the devices off), pretending usage by the user, and
resistance to devices that were forced on them [163].

In this section, we learned that smart speakers are popular devices in the
users’ homes and are mainly used to execute specific tasks or for automation
purposes [16]. Previous work observed that users prefer to enter non-private data
to smart speakers [51, 120]. Previous work also observed that the usage of smart
speakers decreases over time [169]. Reasons are barriers to a missing mental
model of how smart speakers work, resulting in dissatisfaction. However, other
users humanize smart speakers. Therefore, some researchers envision that smart
speakers could act as digital assistants in the future [133]. It remains unclear
whether smart speakers can be used to inform their users about everyday home
information. Furthermnore, investigation needs to be done on how smart speakers
can be designed to fit the users’ routine (RQ6).

42 2 | Background



2.4 Ambient information systems

Ambient information systems display non-urgent information in the periphery
of the user’s attention using abstract and aesthetic displays [107]. Such systems
can either be integrated into existing objects, e.g., by using augmentation or use
additional devices to display information in the surroundings [180]. Since these
devices are visible in the users’ environments, aesthetic aspects are important for
their acceptance [153]. Ambient information systems can use visual [80, 227],
auditory [4], tactile [152] or olfactory [19, 22] cues to deliver information.

Previous work investigated how ambient information systems should display
information [111, 113]. Matthews et al. found that displayed information in ambi-
ent information systems should be perceivable at a glance [111]. The information
displayed by an ambient information system is usually non-urgent but is still
important for the users’ awareness or the sense of the users’ well-being [153].
Therefore, the information should be displayed unobtrusively and in an abstract
way; for example, an ambient information system could use ambient light dis-
plays to present color encoded information. Matviienko et al. analyzed the
color-coding from diverse applications using ambient light displays and suggested
using common metaphors to display information, e.g., the traffic light’s color
pattern [113].

2.4.1 Taxonomies for ambient information systems

Matthews et al. created the first taxonomy of ambient information systems
consisting of abstraction, notification, and transition level. They showed that
optimal information representation depends on the information’s importance and
how much attention the user needs to spend [111]. The information that should be
displayed using an ambient information system can be gained by either extracting
specific features or reducing the information’s fidelity. An ambient information
system’s notification level describes the level of importance of the displayed
information [111]. The notification level contains the items: ignore, change-blind,
make aware, interrupt, and demand attention. Depending on the importance of the
information, the users can ignore it, or the system should make them aware of the
information. For important or urgent information, the system should interrupt the

2.4 | Ambient information systems 43



users from their current primary tasks. The transition describes any permutation
from one notification level to another, e.g., when the displayed information’s
notification level switches from ignoring to interrupt. How such a transition
should be displayed to a user is not described in the taxonomy.

Based on the first taxonomy of Matthews et al. [111], Pousman and Stasko
developed a second taxonomy by analyzing published ambient information sys-
tem from former work [153]. Their taxonomy contains the levels: information
capacity, notification level, representational fidelity, and aesthetic emphasis. The
information capacity describes whether ambient information systems present
information from a single source of data - such a system would be classified
as having a low information capacity or if the ambient information system can
present data from multiple sources - such an ambient information system would
be classified as having a high information capacity. For the notification level,
Pousman and Stasko revised the notification levels from Matthews et al. [111] by
replacing the item ignore with user-poll resulting in the following items: user-poll,
change-blind, make-aware, interrupt and demand attention. The representational
fidelity is similar to the abstraction level in the taxonomy of Matthews et al. [111]
and describes how the presented information is encoded. For their taxonomy,
Pousman and Stasko [153] divided the categories indexical, iconic, and symbolic
into the following items for the representational fidelity: indexical, iconic with
drawings, doodles or caricatures, iconic using metaphors, symbolic using lan-
guage symbols, and symbolic using abstract symbols. The aesthetic emphasis
level of ambient information systems highlights the importance of the aesthetics
of such a display since these displays are usually placed visibly in the environment.
However, an ambient information system’s aesthetics is usually a compromise af-
fected by the information capacity and the representational fidelity of an ambient
information system. Therefore, the aesthetics of an ambient information system
can be accessed by the range from a low to a high aesthetic emphasis.

Similar to Pousman and Stasko [153], Tomitsch et al. also revised the original
taxonomy by Matthews et al. [111] and developed a taxonomy for ambient
information systems [180]. The taxonomy of Tomitsch et al. is more detailed and
contains the following nine levels: abstraction level, transition, notification level,
temporal gradient, representation, modality, source, privacy as well as dynamic of

44 2 | Background



input. The abstraction level describes to which degree the information is abstracted
for the representation. The metric for the degree of abstraction applied is low,
medium-high. The transition describes whether an ambient information system
displays changes in the presented information by applying slow, medium, or fast
transitions. Using slow transitions, users will only be aware of large changes
in the presented data, while applying medium changes will be perceived more
abruptly. Fast changes appear immediately on display. The taxonomy of Tomitsch
et al. uses the same items for the notification level as Matthews et al. [111],
resulting in: ignore, change-blind, make aware, interrupt and demand attention.
The level temporal gradient describes whether an ambient information system
displays only current data or also historical data. The representation of an ambient
display specifies if the display is either an own devices build solely to present
the information (i.e., physical representation) or if the presented information
is integrated into an already existing object (i.e., integrated representation). If
established screens such as Liquid Crystal Displays (LCDs) are used to represent
the information, this belongs to the item 2D. The modality describes which
modality is used to display the information. Possible modalities are: visual,
tactile, olfactory, auditory, or by enabling movement. The source’s level describes
whether information collected in the same environment (i.e., local) is displayed
or if the information was collected at a distant location or if the information
was collected in the virtual world (i.e., virtual). The privacy level describes if
the display is placed in a private, semi-public, or public environment. The level
dynamic of input describes how often the displayed information is updated. The
metric for this level is slow, medium, fast.

2.4.2 Evaluation of ambient information systems

Former work investigated how ambient information systems can be evaluated [81,
82, 107]. One option for the evaluation of prototypes is to conduct an inspection
that is called heuristic evaluation. A heuristic evaluation is used to investigate
whether the system is compatible with the intended needs and preferences of the
users [132]. During a heuristic evaluation of a system, the system is accessed
by a small set of evaluators who judge the system’s compliance with recognized
usability principles (i.e., the heuristics used in the inspection) [132]. Evaluators

2.4 | Ambient information systems 45



of such a system can be usability professionals as well as non-usability spe-
cialists [131]. However, Nielsen found that usability professionals are better at
conducting heuristic evaluations and detecting usability issues than non-usability
specialists [131]. Mankoff et al. improved Nielson’s general heuristic [132] by
revising them to evaluate ambient displays [107]. Hereby, they proposed to also
consider the information design, the information mapping, the visualization of
the states, the usefulness of the information, the easy transition to more detailed
information, and the ’peripherality’ of the ambient display (i.e., the character-
istic of being unobtrusive on one hand but still offering the option to be easily
monitored by the user if necessary). However, an heuristic evaluation can only
identify usability issues connected to the applied heuristic during the inspection;
all other possible usability issues, including issues related to the context of use,
will remain undetected [183].

Hazlewood et al. showed that it is necessary to evaluate ambient displays
outside the lab, e.g., by conducting long-term in-situ studies [81, 82]. Furthermore,
Hazlewood et al. identified four design directions to improve the evaluation of
ambient displays [81]. Research can trigger artificial events during an in-situ
study to address the lack of rarely occurring events in a study. Furthermore,
they can log and analyze when the users are looking at an ambient display, e.g.,
through gaze-detection using an integrated eye-tracker. Another opportunity to
improve ambient displays’ evaluation is to develop multiple ambient displays and
investigate which of them have a potential for sustained use. Finally, to improve
the design of an ambient display, an interaction criticism phase could be added
into an interaction design process.

In this section, we learned that ambient information systems are used to display
information in the users’ periphery. Based on different factors of an ambient
information system, it can influence how the data is perceived, for example, by
implementing a fast change in the displayed data [180]. An important factor for
ambient information systems is their aesthetics. For example, ambient lighting
can be used to display color-coded information to the users [113]. Different
notification levels can be used for ambient information systems to display the
information, e.g., an ambient information system could only display data based
on the user’s request, or the ambient information system can make the user

46 2 | Background



aware of the information or even interrupt the users in their current activities
or demand their attention [153]. However, it remains unclear whether ambient
information systems should be used to represent everyday home information and
which notification/transition levels would fit the users’ routines.

2.5 Mobile Notifications

Mobile notifications are an established communication method to inform users
proactively about different kinds of information. Mobile notifications could
also be used to convey everyday home information. Nowadays, apps inform
users proactively through mobile notifications using visual, auditory, or tactile
cues [88]. Former work analyzed which kind of notifications users receive on
their smartphones [147, 161]. Pielot et al. found in an in-situ study in 2014 that
their participants received about 63.5 notifications per day [147]. Notifications on
smartphones inform their users mainly to support communication [147]. Users
value notifications from messaging apps and notifications containing information
about people or their current context [161]. Weber et al. found that users prefer
receiving notifications on their smartphones - although they are used to receive
notifications on all of their smart devices, including smartwatches and tablets
[213]. However, the proximity to devices, if they are currently used, and the
user’s current location can affect if users are willing to receive notifications on
their devices.

In the following subsections, we will investigate how notifications are per-
ceived, what kind of adverse effects can be caused by notifications, which effects
are caused by mobile unavailability, and which approaches researchers evaluated
to reduce the adverse effects caused by notifications.

2.5.1 Awareness and perception of notifications

Chang et al. investigated the perception of mobile notifications. They found that
only 62% of the notifications received were seen by the users [27]. Weber et
al. revealed that the users perceive the majority of the incoming notifications as
not important and non-urgent [210]. Other works investigate the attentiveness
of notifications [9, 46, 60, 147]. Pielot et al. extracted features to generate a

2.5 | Mobile Notifications 47



model that predicts whether a user will see a message within the next minutes
or not [147]. However, Dingler et al. found that inattentiveness occurs rarely
and subsides quickly [46]. Bahir et al. showed that users were more attentive
regarding notifications that contained either images/icons or action buttons that
enable them to respond to the notification within the notification drawer [9]. Bahir
et al. observed that their participants responded faster to notifications, which were
received in the afternoon or evening [9]. In contrast, Fisher et al. found that a
notification’s content affects how fast users attend to notifications [60].

Prior work also investigated how users deal with the number of incoming
notifications [191, 208]. Weber et al. found that there are three different kinds
of users [208]: (1) The frequent cleaners who frequently attend to incoming
notifications, (2) the notification regulators who respond to notifications before
the number gets too high, (3) the notification hoarders who usually do not dismiss
notifications regularly. Voit et al. found that only a few users configure the
notification system on their smartphones to disable notifications [191]. Instead,
they observed that users apply other strategies to deal with notifications, including
ignoring them or uninstalling the application, muting the smartphones, or even
putting their smartphones away.

Further, Exler et al. showed that notifications displayed using tactile or au-
ditory feedback were most perceptible [56]. However, auditory notifications
were perceived as too annoying, disturbing, and obtrusive for everyday use [56].
This confirms the observation of Gallud et al. which stated users are switching
from receiving notifications with sound to visual notifications [64]. Tactile no-
tifications were perceived as more private and subtle; however, this can lead to
awkward situations when others cannot foresee an action arising from such a
notification [76].

2.5.2 Negative effects of notifications

A body of related work investigated which negative effects such as distractions,
interruptions, lower productivity and higher error-prone performance are caused
by notifications [11, 88]. Weber et al. showed that users underestimate the num-
ber of notifications that they receive in their daily lives [215]. Iqbal and Horvitz
found that email notifications displayed on desktop computers cause distractions

48 2 | Background



from primary tasks at work [88]. Mehrotra et al. showed how disruptive the
users perceive a notification depends on how the notification is displayed, the
sender-recipient-relationship, and the primary task in which the user is currently
engaged [117]. In addition, Bailey and Iqbal observed that interruptions in mo-
ments with increasing mental workload also cause negative effects including a
slower task performance and frustrations [11]. Turning off notifications [145]
and blocking non-work related distractions from social media [108] lead to in-
creased productivity and reduced distractions. However, users feel less responsive
and less connected to their social contacts [145]. In addition, some users feel
when receiving notifications more temporal demand and stress [108]. Pielot et
al. showed that users experience social pressure to respond fast to incoming
messages [147]. Users are feeling a social obligation to answer fast to incoming
messages as otherwise other persons such as family members or friends express
frustrations regarding their delayed and unpredictable answer patterns [6].

Previous work also investigated how notifications are related to negative
behavior patterns [6, 103]. Aranda and Biag found that users are triggered by
incoming notifications or phantom cues to use their devices [6]. Afterward, users
are keeping in the loop of interacting with the device or application, e.g., by
implemented automatic triggers within websites or applications such as infinite
scrolling or recommended content [6].. This is especially problematic since Lee et
al. showed that notifications could initiate problematic usage patterns especially
for users that are more susceptible for smartphone overuse [103].

2.5.3 Effects of mobile unavailabilty

Former work also investigated how users experience mobile unavailability [6,
160]. Aranda and Biag found that the level of control (i.e., voluntarily or forced
disconnect) and the duration of non-use (i.e., short-term or long-term) are affecting
how the disconnection is experienced by their users [6]. When their participants
self-consistently disconnected to draw boundaries for their devices usage, they
experienced the "joy-of-missing-out" (i.e, they were happy and less stressed).
However, participants that were disconnected because of even short-term outages
experienced the "fear-of-missing-out" (i.e., they felt anxious and inconvenient).
Russo et al. analyzed comments on a web article and found users spend time

2.5 | Mobile Notifications 49



and effort deciding when to disconnect from their devices [160]. Further, they
identified four main reasons why users self-consistently disconnect from their
devices; (1) to improve their current role performance, this includes focusing on
their current primary activities as well as resting and recovering, (2) to implement
a personal digital philosophy, e.g., to be an example for others such as children
(3) to minimize undesirable behaviors, e.g., to not interact with their phones while
being out with friends, and (4) to shield their own priorities within their lives.

2.5.4 Reducing negative effects caused by notifications

Another strand of prior work aimed to reduce distractions by developing models
for receiving notifications or delaying incoming notifications to opportune mo-
ments. A body of work investigated delaying incoming notifications to opportune
moments such as breaks between different primary tasks [59, 146] and identi-
fying opportune moments for interruptions. Mehrotra et al. showed that taking
the sender-recipient relationship, the context, as well as the current context into
account, leads to a better prediction of the users’ interruptibility [116]. Adamczyk
et al. showed that identifying opportune moments in a user’s task sequence can
decrease the negative effects caused by interruptions on the social attribution
and the user’s emotional state [2]. In contrast to other work, Weber et al. found
that users mainly delayed notifications related to people and events that were not
fitting to their daily routines [217]. Further, they found that notifications should
no longer be delayed than the following morning. In order to detect opportune
moments, Okoshi et al. developed a system that detects breakpoints in the current
activity on the users’ smartphones [139] as well as breakpoints between physical
activities of the user [138].

Other work investigated using rules to reduce the number of notifications
received [7, 115]. [7] Mehrotra et al. developed a machine-learning model that
learns users’ preferences for receiving notifications [115]. Hereby, the system
generates rules based on the user’s former response to such notifications, the type
and arrival time of notification, as well as the context of the user (i.e., the users’
activity and location). These rules are displayed to the users who can either accept
or reject them.

50 2 | Background



In this section, we found that current applications inform their users proac-
tively using mobile notifications, e.g., to notify the users on their smartphones.
While users value being proactively informed about incoming information [161],
notifications also cause negative effects for the users such as distractions, interrup-
tions, lower productivity, and a higher error-prone performance [11, 88]. A body
of work investigated how the negative effects of incoming notifications could be
reduced, e.g., by delaying the information to opportune moments [59, 145] or by
using rules [7] or machine-learning approaches [115]. Therefore, designers of
smart home appliances that display additional information to the users should
also consider the number of notifications that the users receive in their daily lives
to not overwhelm the users with more information.

2.6 Summary

The introduction of novel technologies in the users’ homes enables new opportu-
nities for smart home appliances to support the users in their daily lives, e.g., by
informing them about everyday home information. Previous research investigated
the acceptance, effects of different modalities for displaying everyday information
in the home context [186, 187, 205, 206]. The investigations of Vastenburg et
al. revealed that presenting more urgent everyday information is more accepted
by the users. Also, low-urgent information should be delayed until the urgency
increased [186]. Further, the modality that is used to deliver the information to
the users affects the time to perceive the information, but not the disruption or
the performance according to the user’s current primary task [206]. Neither does
the current primary task affect the acceptance of receiving notifications about
everyday information [186].

Different technologies can be used to implement smart home reminder systems
that convey everyday home information to users. These technologies include
using ambient information systems, mobile notifications, and smart speakers.
Regardless of which technology is used to convey home information to the
users, former work identified factors that affect the acceptance of those systems,
including visualizing smart home data [118, 119], offering opportunities to adapt
the visualizations [26], and considering the social dynamics in the home [181].

2.6 | Summary 51



Marky et al. found that the display’s location in the home can affect the willingness
to display sensitive data in the home environment. Further, they found that
everyday information, such as household data, can be shared with others. Some of
their participants even stated that it would be beneficial for them if other persons,
including visitors, could see their household data.

Czerwinski et al. envisioned that many appliances would compete for the
users’ attention in the future [43]. Therefore, home information should be dis-
played subtly. However, it remains unclear how smart home appliances should
convey home information to their users. For example, an investigation of suitable
modalities (RQ3) and locations (RQ4) to present everyday home information is
missing. Besides, we need to investigate how the information should be displayed
(RQ5). For example, a study should investigate whether smart home appliances
should persistently visualize their current state or whether the user should only
be informed based on specific events. Further, we need to study how the smart
home appliances displaying everyday information need to be designed to fit into
the users’ routines (RQ6).

In addition to integrating AR applications in domestic environments as
Knierim et al. [95] suggested, novel technologies such as AR can also be used for
rapid prototyping and evaluating early prototypes [97, 151, 188], e.g., for smart
home appliances. However, it remains unclear whether using AR or VR affects on
the results of the conducted study, i.e., if the gained results will be reproducible,
valid, and reliable [85]. To be able to study smart home appliances informing
the users about everyday information, it is crucial to investigate which evaluation
methods are suitable for the evaluation of early prototypes in the context of smart
home appliances presenting everyday information (RQ1) and to understand the
advantages and disadvantages of each evaluation method (RQ2).

52 2 | Background



3
Evaluation Methods for Smart
Home Artifacts

Empirical studies collecting quantitative and qualitative feedback are essential
to investigate how users experience smart home artifacts’ design. Technical
progress constantly enables new study methods that can be used for evaluations of
prototypes. Online surveys, for example, make it possible to collect feedback from
remote users. Progress in augmented reality (AR) and virtual reality (VR) enables
us to collect feedback with early designs. In-situ studies enable researchers
to gather feedback in natural environments. While these methods have unique
advantages and disadvantages, it is unclear if and how using a specific method
might affect the results and, therefore, have effects while applying the user-
centered design process [135]. Therefore, in this chapter, we will investigate
which evaluation methods best suits the evaluation of smart home appliances that
inform the user about non-urgent everyday information (RQ1). Further, we will
need to understand the advantages and disadvantages of the different evaluation
methods to choose the right method for evaluating research probes (RQ2).

53



In detail, we report about a study with 60 participants to compare the different
evaluation methods online survey, a study using VR, a study using AR, a study
using a traditional lab setup, and evaluating the prototypes in the users’ homes
(i.e., in-situ study) for the evaluation of early prototypes (see also Figure 3.1).

This chapter is based on the following publication :

A. Voit, S. Mayer, V. Schwind, and N. Henze. “Online, VR, AR, Lab, and In-Situ: Compari-
son of Research Methods to Evaluate Smart Artifacts.” In: Proceedings of the of the 2019
CHI Conference on Human Factors in Computing Systems. CHI 19. New York, NY, USA:
ACM, 2019, p. 12

3.1 Related work

Our work is inspired by previous work that applied and investigated different
study methods. It is based on a body of work that compared multiple methods to
reveal how the study methods can affect the results of a study.

3.1.1 Empirical Methods in HCI

A range of methods is widely used to evaluate prototypes. Among the most
established methods are online surveys [142, 198], lab studies [3, 22, 193], and in-
situ studies [83, 119, 188]. Advances in technology further enable new methods;
recent examples include using VR [95] and AR [151] to evaluate prototypes.

Online surveys are the most efficient opportunity to conduct surveys with a
broad range of participants as they are cheap and time efficient [44, 174]. Further,
online surveys are comfortable for participants because they can attend the survey
when they are available and at home [44].

Lab studies are used to evaluate prototypes in a controlled setting without
interruptions [48]. In lab studies, a research assistant acts as a human moderator
to gain results with a high internal validity. Lab studies can occur either in an
abstract setting [50] or in environments that resemble parts of the real world to
simulate a natural usage context [93, 176].

54 3 | Evaluation Methods for Smart Home Artifacts



(a) Online (b) VR

(c) AR (d) Lab

(e) In-Situ

Figure 3.1: Examples of the five study methods online, virtual reality (VR) and aug-

mented reality (AR), lab study and in-situ studies.

3.1 | Related work 55



In contrast to online surveys and lab studies, in-situ studies are used to
evaluate prototypes in their natural environment [48, 157], e.g., at home, to
determine results with high external validity [83]. In addition, in-situ studies can
be used to understand the user experience [24, 157] and capturing the context
of use [158], e.g., by combining different data collection strategies such as
interviews and logging data in the background. However, by conducting in-situ
studies, researchers are not fully in control over the environment. Therefore,
distractions and interruptions, e.g., caused by other persons, can occur [48].

Advances in technology, enable to conduct studies using VR and AR to
evaluate prototypes. Especially, VR can be useful to conduct studies that are too
expensive or too dangerous to be conducted in the real world or the lab [48, 49].
VR studies can be conducted outside of the lab and even with a large number
of participants over longer periods of time [121]. Former research compared
different presentation formats for VR studies and found that using head-mounted
displays provides the most immersive experience [32]. Researchers also started
using AR for rapid prototyping and the evaluation of radically new interfaces [97,
151].

When conducting an evaluation, it is essential that the evaluation investigates
not only the prototypes’ usability aspects, but considers also hedonic and emo-
tional aspects of the interaction [184]. When investigating the user experience of
prototypes, Väänänen-Vainio-Mattila et al. found that a majority of former work
investigates the first-time experience with the prototypes [184].

3.1.2 Comparison of Empirical Methods

Previous work compared the effects of conducting online surveys or lab studies
on the participants and the study results [30, 44]. Online surveys have higher
dropout rates as participants in the lab feel more committed to participate in the
experiment [44]. Further, lab study participants can be more engaged and can
also be more accurate when solving demanding tasks than in online surveys [44].
One reason is that participants in online surveys are more distracted than those in
lab studies [30]. In online surveys, researchers are not present and therefore have
no control over the environment where the survey is answered [30, 44].

56 3 | Evaluation Methods for Smart Home Artifacts



A large body of work compared lab and in-situ studies. There is an ongoing
discussion whether it is worth the hassle to conduct in-situ studies to evaluate
prototypes [92, 93, 130, 158]. Most of these comparisons showed that both
evaluation methods enable users to identify similar usability problems [86, 90,
93]. However, other studies found that themes related to usability problems (i.e.,
cognitive load and interaction style) identified in the in-situ study, were not found
in the lab study [130]. Further, in-situ studies enable finding usability problems
associated with external factors of the natural environment that are difficult to
simulate in regular lab studies, e.g., the movement in a train [50]. In addition, Sun
and May found differences in participants’ engagement [176]. They collected
more feedback related to data validity and precision in the in-situ study, while in
the lab participants focused more on details of the interface.

Related work also investigated differences in the perceived user experi-
ence [157, 176]. It has been found that the surroundings of a study can affect the
user experience. For example, Sun and May found that the user experience ratings
in the in-situ study were higher as participants were affected by the positive
atmosphere in a sports stadium [176].

Former studies differed in their setups’ level of realism. Some studies were
conducted in highly realistic lab setups that resembled parts of the natural en-
vironments [93, 176] for comparison. Other comparisons, were conducted in
more abstract lab setups, e.g. an actual train ride was compared with sitting at a
table [50]. Some in-situ studies were conducted in the actual context such as a
sports stadium [176], while others were conducted in similar environments which
the researchers could better control [130].

Finally, Kjeldskov reported that the suitability of a method (i.e., lab or in-
situ study) depends on the specific research questions and goals [92]. However,
previous work agrees that in-situ studies are better suited to investigate how a
prototype integrates into users’ lives, to capture the real user behavior and to
determine the context of use with high external validity [86, 158].

3.1.3 Summary

HCI research uses different study methods with different advantages and dis-
advantages for the evaluation of prototypes. A body of work investigated how

3.1 | Related work 57



different methods (i.e., online survey vs. lab [30, 44] and lab vs. in-situ [92, 93,
130, 158]) affect the results of usability and user experience investigation. Which
study method is the best method and should be applied depends on the research
questions. For example, in-situ studies should be conducted to investigate the
integration of a prototype into the participants’ daily lives or obser