Empirical Assessment and Improvement of Ubiquitous Notifications Von der Fakultät für Informatik, Elektrotechnik und Informationstechnik und dem Stuttgart Research Centre for Simulation Technology der Universität Stuttgart zur Erlangung der Würde eines Doktors der Naturwissenschaften (Dr. rer. nat.) genehmigte Abhandlung Vorgelegt von Dominik Christian Weber aus Backnang Hauptberichter: Prof. Dr. Niels Henze Mitberichter: Prof. Dr. Christian Becker Mitberichter: Prof. Dr. Stefan Schneegass Tag der mündlichen Prüfung: 27.10.2023 Institut für Visualisierung und Interaktive Systeme der Universität Stuttgart 2023 Zusammenfassung Intelligente Geräte sind allgegenwärtig geworden. Geräte wie Smartphones, Smartwatches, Tablets, Laptops und Smart-TVs begleiten uns den ganzen Tag. Dank Fortschritten bei der Rechenleistung und Drahtlostechnologien sind diese Geräte immer eingeschaltet und immer verbunden. Während einige Geräte nur situativ genutzt werden, sind andere Geräte wie Smartphones immer beim Nutzer. Dies hat die Art und Weise, wie mit diesen Geräten interagiert wird, grundlegend verändert. Anstatt manuell nach Neuigkeiten und neuen Nachrichten zu suchen, können uns diese Geräte rund um die Uhr proaktiv durch Benachrichtigungen über neue Ereignisse informieren. Von neuen Nachrichten über Erinnerungen bis hin zu Systemaktualisierungen - Benachrichtigungen sind grundsätzlich persönlich und decken ein breites Spektrum an Themen ab. Während Benachrichtigungen von den Nutzern geschätzt werden und ihnen das Gefühl geben, verbunden zu sein, können sie auch zu Unterbrechungen und Ablenkungen führen. Da immer mehr Dienste auf immer mehr Geräten auf Benachrichtigungen zurückgreifen, werden die potenziellen negativen Auswirkungen von Benachrichtigungen verstärkt. So kann beispielsweise eine einzige E-Mail einen Benutzer auf mehreren Geräten mit verschiedenen Modalitäten benachrichtigen. Um negative Auswirkungen zu verringern, ist ein Verständnis der verschiedenen Kategorien von Benachrichti- 3 gungen, der verschiedenen Geräte und der Bedürfnisse der Benutzer erforderlich. Das Benachrichtigungsmanagement ist ein Balanceakt zwischen dem Stillen des Informationsbedürfnisses der Benutzer und der Wahrung ihrer Aufmerksamkeit. Diese Arbeit beschäftigt sich mit der empirischen Bewertung und Verbes- serung von allgegenwärtigen Benachrichtigungen. Es werden mehrere Nutzer- studien präsentiert, von Online-Umfragen, Laborstudien, In-situ-Studien bis hin zu großangelegten In-the-Wild-Studien. Zunächst wird auf die Bewertung und das Management mobiler Benachrichtigungen auf Smartphones eingegangen und anschließend auf die Herausforderungen der Durchführung von kontrollier- ten In-situ- und In-the-wild-Studien unter Wahrung der Privatsphäre der Nutzer. Es wird ein Benachrichtigungsdatensatz präsentiert, verschiedene Nutzertypen vorgeschlagen und neue Ansätze vorgestellt, mit denen die Nutzer über ihre Benachrichtigungen reflektieren und sie verwalten können. Anschließend wird der Anwendungsbereich auf andere Gerätetypen wie Smartwatches, Tablets und Laptops erweitert, um ein ganzheitliches Verständnis dafür zu schaffen, wie sich diese Geräte in Bezug auf die Erwartungen der Nutzer an den Empfang von Benachrichtigungen unterscheiden, indem Aktivitätsprotokollierungen auf mehre- ren Geräten mit Erfahrungsstichproben kombiniert werden. Schließlich wird der Anwendungsbereich erneut erweitert, um große und allgegenwärtige Displays einzubeziehen. Abschließend wird ein Open-Source-Protokollierungsframework für mobile Geräte vorgestellt, damit andere Entwickler und Forscher auf dieser Arbeit aufbauen können. Der Beitrag dieser Arbeit besteht aus drei Teilen. Erstens werden in dieser Arbeit mehrere Ansätze zur Erforschung allgegenwärtiger Benachrichtigungen vorgestellt, von kontrollierten Laborstudien bis hin zu groß angelegten Studien in freier Wildbahn. Zweitens bietet die Arbeit Einblicke in die Benachrichtigungsprä- ferenzen und -interaktionen der Nutzer auf verschiedenen Gerätetypen. Drittens wird ein technischer Beitrag geleistet, der ein Open-Source-Framework zur Proto- kollierung von Benachrichtigungen und einen Datensatz für Benachrichtigungen umfasst. Diese Beiträge bilden eine Grundlage für die zukünftige Forschung zu allgegenwärtigen Benachrichtigungen. 4 Abstract Smart devices have become ubiquitous. Devices like smartphones, smartwatches, tablets, laptops, and smart TVs accompany us throughout the day. Advancements in computational efficiency and wireless technologies allow these devices to be always on and always connected. While some devices are used situationally, other devices like smartphones are always with the user. This inherently changed how we interact with these devices. Instead of manually looking for news and new messages, these devices can proactively inform us about new events through notifications around the clock. From new messages, reminders, to system updates, notifications are fundamentally personal and cover a wide range of topics. While notifications are valued by users and make them feel connected, they can also cause interruptions and distractions. With more and more services making use of notifications on more and more devices, potential adverse effects are amplified. For instance, a single email might alert a user on multiple devices using multiple modalities. To reduce adverse effects, an understanding of different categories of notifications, different devices, and user needs is required. Notification manage- ment is a balancing act between satisfying users’ information needs and respecting their attention. This thesis investigates the empirical assessment and improvements of ubiq- uitous notifications. We present multiple user studies, from online surveys, lab studies, in-situ studies to large-scale in-the-wild studies. We first focus on the 5 assessment and management of mobile notifications on smartphones, tackling the challenges of conducting in-situ controlled and in-the-wild user studies while preserving the users’ privacy. We present a notification data set, propose user types, and introduce new approaches for users to reflect on and manage their notifications. We then expand the scope to include other device types such as smartwatches, tablets, and laptops to create a holistic understanding of how these devices differ regarding user expectations for receiving notifications by combin- ing activity logging on multiple devices with experience sampling. Afterward, we expand the scope again to include large and pervasive displays. Finally, we present an open-source logging framework for mobile devices to enable other developers and researchers to build on top of this work. The contribution of this thesis is threefold. First, this thesis introduces multiple approaches to conducting research on ubiquitous notifications, from controlled lab studies to large-scale in-the-wild studies. Second, the thesis provides insights into users’ notification preferences and interactions on different types of devices. Third, a technical contribution, including an open-source notification logging framework and notification data set. These contributions are a foundation for future research on ubiquitous notifications. 6 Acknowledgements This thesis would not have been possible without the support of some truly awe- some people. First I want to thank my supervisor Niels Henze, who guided me from being a student assistant, through my diploma thesis, to this PhD thesis. Thank you for your trust and for giving me the freedom to follow my interests. A little less freedom might have sped things up but here we are. Second I want to thank the PhD committee, consisting of Christian Becker, Stefan Schneegass, and Ralf Küsters for the interesting discussions and smooth process. Special thanks to Albrecht Schmidt for creating a truly unique environment at the Uni- versity of Stuttgart where everyone could feel empowered. Thank you Alireza Sahami Shirazi for setting me on this path. Alexandra Voit, thank you for the close collaboration and support throughout these years and after. Sven Mayer, thank you for the motivation and pushes. Rufat Rzayev, thank you for sharing many hotel rooms even though some had bugs. Valentin Schwind, thank you for the nightmares about uncanny cats. Huy Viet Le, thank you for enduring me as an office mate. Passant El.Agroudy and Sarah Faltaous, thank you for the shared suffering and support. Yomna Abdelrahman, thank you for your giant heart and delicious food. Special thanks to Jakob Karolus, Markus Funk, Pascal Knierim, and Thomas Kosch for the dedicated WhatsApp group and “When do you submit?” pings after every single conference. It helped. 7 Shoutout to the hciLab group at the University of Stuttgart: Bastian Pfleging, Céline Coutrix, Florian Alt, Francisco Kiss, Hyunyoung Kim, Katrin Wolf, Lars Lischke, Lewis Chuang, Mariam Hassib, Matthias Hoppe, Mauro Avila, Miriam Greis, Nitesh Goyal, Norman Pohl, Patrick Bader, Paweł W. Woź- niak, Romina Poguntke, Thomas Kubitza, Tilman Dingler, and Tonja Machulla. It was great having all of you around. Every one of you helped to create a truly special place. Extended greetings to the VIS(US) group and friends: Dominik Herr, Edwin Püttmann, Kuno Kurzhals, and Tanja Blascheck. At SimTech, thank you, Andrea Barth, for co-advising my milestone presentation. Reiner Dietz, thank you for granting me root access. Thanks to Anja Mebus, Murielle Naud-Barthelmeß, and Eugenia Komnik for your administrative support. Thanks to all the students I worked with. Espcially David Hägele, Frank Bastian, Gisela Kollotzek, Lucas van der Vekens, Marcus Hepting, Marvin Tiedtke, Philipp Kratzer, and Rodrigo Ventura Fierro whose contributions ended up in this thesis. And Jonas Auda who ended up finishing his PhD before me. Great work! If I missed anyone, I would like to sincerely apologize. I am writing these words several years after concluding the actual work of the thesis, so some things already started to slip from my mind. As a catch-all, I want to thank everyone I worked with at the University of Stuttgart and in research projects. Thanks to everyone who inspired me and provided feedback at conferences, doctoral colloquia, and workshops. Finally, I want to thank my parents Margot and Bernhard Weber for their endless love and support throughout this and everything else in life. Thank you. 8 9 Table of Contents 1 Introduction 15 1.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.2 Challenges and Research Contributions . . . . . . . . . . . . . . . . 18 1.3 Methodology and Evaluation . . . . . . . . . . . . . . . . . . . . . . 20 1.4 Research Context . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.4.1 Graduate School . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.4.2 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.4.3 Funding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 1.4.4 Collaborations . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.5 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2 Background and Related Work 27 2.1 Notification Definition . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.2 Notification Systems on Different Devices . . . . . . . . . . . . . . . 28 2.2.1 Smartphones . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.2.2 Tablets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.2.3 Smartwartches . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.2.4 Desktop PCs and Laptops . . . . . . . . . . . . . . . . . . . . 31 2.3 Prior Work by the Author . . . . . . . . . . . . . . . . . . . . . . . . 31 2.3.1 Researching Mobile Notifications at Scale . . . . . . . . . . . . 32 11 2.3.2 Expanding to Multi-Device Environments . . . . . . . . . . . . . 34 2.3.3 Limitations and Learnings . . . . . . . . . . . . . . . . . . . . 35 2.4 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.4.1 Mobile Notifications . . . . . . . . . . . . . . . . . . . . . . . 36 2.4.2 Beyond Mobile . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.4.3 Interruptions and Adverse Effects . . . . . . . . . . . . . . . . 42 2.4.4 Notification Management . . . . . . . . . . . . . . . . . . . . . 43 2.4.5 Research in the Wild . . . . . . . . . . . . . . . . . . . . . . . 47 2.4.6 App Stores and External Validity . . . . . . . . . . . . . . . . . 49 2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3 Notifications on Mobile Devices 51 3.1 Mobile Notification Drawers . . . . . . . . . . . . . . . . . . . . . . 52 3.1.1 Notifications on Android . . . . . . . . . . . . . . . . . . . . . 53 3.1.2 Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.1.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.1.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 3.1.5 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 3.1.6 Open-Source Data Set . . . . . . . . . . . . . . . . . . . . . . 72 3.2 Annotating Mobile Notifications . . . . . . . . . . . . . . . . . . . . . 72 3.2.1 Experience Sampling of Mobile Notifications . . . . . . . . . . . 73 3.2.2 The “Annotif” Annotation System . . . . . . . . . . . . . . . . . 74 3.2.3 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 3.2.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 3.2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 3.2.6 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 3.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 4 Managing Mobile Notifications 97 4.1 Reflecting on Mobile Notifications . . . . . . . . . . . . . . . . . . . 98 4.1.1 Notification Dashboard . . . . . . . . . . . . . . . . . . . . . . 98 4.1.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.1.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 12 4.2 User-Defined Deferral of Mobile Notifications . . . . . . . . . . . . . . 108 4.2.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 109 4.2.2 System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 4.2.3 In-The-Wild Study . . . . . . . . . . . . . . . . . . . . . . . . 116 4.2.4 Controlled Study . . . . . . . . . . . . . . . . . . . . . . . . . 123 4.2.5 Discussion and Limitations . . . . . . . . . . . . . . . . . . . . 131 4.2.6 Design Implications . . . . . . . . . . . . . . . . . . . . . . . 132 4.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 5 Notifications in Multi-Device Environments 137 5.1 Quantitative Investigation of Notifications in Multi-Device Environments . 138 5.1.1 Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 5.1.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 5.1.3 Discussion and Limitations . . . . . . . . . . . . . . . . . . . . 150 5.1.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 5.2 Qualitative Investigation of Notifications in Multi-Device Environments . . 152 5.2.1 Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 5.2.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 5.2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 5.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 6 Notifications on Large and Pervasive Displays 161 6.1 Notifications on Smart TVs . . . . . . . . . . . . . . . . . . . . . . . 162 6.1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 163 6.1.2 Focus Groups . . . . . . . . . . . . . . . . . . . . . . . . . . 165 6.1.3 Online Survey . . . . . . . . . . . . . . . . . . . . . . . . . . 171 6.1.4 Lab Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 6.1.5 Design Guidelines . . . . . . . . . . . . . . . . . . . . . . . . 186 6.2 Notifications on Public Displays . . . . . . . . . . . . . . . . . . . . 188 6.2.1 System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 6.2.2 Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 6.2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 6.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 13 7 Notification Logging Framework 201 7.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 7.1.1 Data Sources . . . . . . . . . . . . . . . . . . . . . . . . . . 202 7.1.2 Data Consolidation . . . . . . . . . . . . . . . . . . . . . . . 204 7.1.3 Data Persistence . . . . . . . . . . . . . . . . . . . . . . . . . 205 7.1.4 Data Processing . . . . . . . . . . . . . . . . . . . . . . . . . 205 7.1.5 Extensibility . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 7.2 Application Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . 205 7.2.1 Record and Replay of Notifications . . . . . . . . . . . . . . . . 206 7.2.2 Reflection on Mobile Notifications . . . . . . . . . . . . . . . . 206 7.2.3 Integration in Existing Infrastructures . . . . . . . . . . . . . . . 206 7.2.4 Novel Experiences . . . . . . . . . . . . . . . . . . . . . . . . 207 7.3 Open-Source Framework . . . . . . . . . . . . . . . . . . . . . . . . 207 8 Conclusion and Future Work 209 8.1 Summary of Research Contributions . . . . . . . . . . . . . . . . . . 211 8.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 Bibliography 219 List of Figures 247 List of Tables 251 List of Acronyms 253 14 1 Introduction In the 1991 article “The Computer for the 21st Century,” Mark Weiser outlined the vision of ubiquitous computers [186]. He predicted that future computers would be integrated seamlessly into the world and “vanish into the background.” According to Weiser, these ubiquitous computers would come in different sizes: Tabs (“inch-scale”), pads (“foot-scale”), and boards (“yard-scale”). Although Weiser mentioned several challenges in terms of software and hardware require- ments, including wireless connectivity, he highlighted the benefits of ubiquitous computers: “Most important, ubiquitous computers will help overcome the prob- lem of information overload. There is more information available at our fingertips during a walk in the woods than in any computer system, yet people find a walk among trees relaxing and computers frustrating. Machines that fit the human environment, instead of forc- ing humans to enter theirs, will make using a computer as refreshing as taking a walk in the woods.” (Mark Weiser, 1991 [186]) Thirty years later, computers have indeed become ubiquitous. Mobile phones gained rapid adoption and evolved into always-connected smartphones that ac- 15 company users throughout the day. Smartwatches emerged as companion devices to smartphones and standalone wrist-worn computers. Tablet-computers comple- ment desktop PCs and laptops. Televisions in the living room have become smart as well, allowing access to media on demand. Outside the home, public displays provide users with information. These smart devices are often connected using high-speed and highly-efficient wireless networks. Not only are these devices providing users with information at their fin- gertips. Smart devices can provide users with information proactively. Using visual, tactile, and auditory cues, these devices can use notifications to gain the user’s attention [66]. Notifications can be issued for all kinds of events, from communication-related, to breaking news, weather updates, to system alerts. However, prior work found that while users value such notifications, they can also cause interruptions and distractions [47, 67, 137]. This can lead to adverse effects, such as increased stress [193], inattention [78], and reduced task per- formance [30]. In an environment with ubiquitous computing, devices these adverse effects might become amplified. For instance, in an environment with a smartphone, smartwatch, tablet, and laptop, a single email might cause all devices to notify the user. As ubiquitous computing environments expand, we need to research how to balance the users’ information needs while respecting their attention. 1.1 Research Questions This thesis investigates the empirical assessment and improvement of ubiquitous notifications. As this is a broad topic, we first need to define the scope of the thesis. Smartphones themselves have become ubiquitous with a high market penetration. Combined with the fact that smartphones are typically with the user throughout the day, we focus on mobile notifications in the first part of this thesis. We then expand the scope beyond mobile notifications to include different kinds of personal devices, such as smartwatches, tablet computers, and desktop PCs/ laptops. Finally, we expand the scope further to include large and pervasive displays, such as smart TVs and public displays. 16 1 | Introduction The research questions (RQs) are listed in Table 1.1. The first research ques- tion (RQ1) is about how mobile notifications materialize on smartphones. While prior work already investigated how many notifications users receive on a daily basis, which kinds of notifications are valued by users, and how fast users attend notifications, what is missing is an assessment of how many pending notifications users see throughout the day and whether there are different approaches in at- tending those notifications. Assessing notifications in user studies is challenging from a privacy perspective since notifications are inherently personal. Our second research question (RQ2) is, therefore, how can we assess notifications, includ- ing the content, while respecting the privacy of participants. Apart from purely assessing mobile notifications, another aspect is assisting users by improving mobile notification management. The third research question (RQ3) is about novel approaches to improve the state-of-the-art of notification management. Looking beyond mobile notifications on smartphones, an open research ques- tion is the differences between different kinds of devices when it comes to whether users would like to receive notifications on those devices. Our fourth research question (RQ4) is about how various types of personal devices differ in multi- device environments with regard to receiving notifications. Further, we differ- entiate between notifications on personal devices and on devices that are often shared, such as smart TVs. The fifth research question (RQ5) is about consid- erations for showing notifications on such devices. Finally, going even further beyond an open research question is how public displays can be used to display personal notifications. Our sixth and final research question (RQ6) is about what to consider when showing highly personal notifications on public displays. 1.1 | Research Questions 17 Research Question No. Chapter Assessing Notifications on Mobile Devices How do notifications materialize on smartphones, and how are users managing them? RQ1 Chapter 3 How can we assess notifications in detail while respecting users’ privacy? RQ2 Chapter 3 Improving the Management of Mobile Notifications How can we support users with managing mobile notifi- cations? RQ3 Chapter 4 Beyond Mobile Notifications How do various types of personal devices differ in multi- device environments with regards to displaying notifica- tions? RQ4 Chapter 5 What are the considerations when displaying notifications on smart TVs? RQ5 Chapter 6 What are the considerations when displaying notifications on public displays? RQ6 Chapter 6 Table 1.1: The research questions that are being addressed in this thesis. 1.2 Challenges and Research Contributions A major challenge is that notifications are highly personal and timely. When researching actual notifications that users receive on a daily basis, lab studies are often not suitable. Users receive notifications throughout the day and these often include sensitive communication-related content. For notification research, we often need to trade the internal validity of user studies for external validity by conducting studies in-situ. However, this poses new challenges, such as heterogeneous device environments, little control, and no supervision. We also have to consider if and how the user studies affect the study results. For instance, prompts for questionnaires are typically implemented as notifications as well. A major challenge of this work is how we can create unobtrusive research probes to answer our research questions. Table 1.2 provides an overview of the research probes and contributions in this thesis. 18 1 | Introduction Picture Description Chapter Assessing Notifications on Mobile Devices 0 500 1000 1500 2000 2500 3000 Snapshots 0 20 40 60 80 100 No tif ica tio ns (G ro up ed ) The Notification Drawer data set contains over 8.8 million notification drawer snapshots from almost 4,000 Android devices. Chapter 3 Annotif is a system that allows users to an- notate notifications and share them with re- searchers in a privacy-respecting manner. Chapter 3 Improving the Management of Mobile Notifications The Notification Dashboard allows users to reflect on the number of notifications they receive on a daily basis. Chapter 4 NHistory is an Android app that allows users to “snooze” notifications for a duration or to a specific point-in-time. Chapter 4 Beyond Mobile Notifications The dedicated ESM device allows triggering questionnaires at random times throughout the day to avoid affecting other devices. Chapter 5 The TV lab study app allows video play- back while overlying previously recorded user-customizable notifications. Chapter 6 PD Notify is a system that allows users to mir- ror their smartphone notifications on nearby public displays. Chapter 6 Notification Logging Framework Notification Log is a notification logging framework for mobile notifications imple- mented as an open-source Android app. Chapter 7 Table 1.2: An overview of the research probes and data sets in this thesis. 1.2 | Challenges and Research Contributions 19 1.3 Methodology and Evaluation Throughout this thesis, we use several methods for data collection. In two cases, published apps in app stores to conduct large-scale in-the-wild studies with hundreds and thousands of users. Further, we conducted smaller scale in-situ studies by asking participants to install apps and following up with semi-structured interviews or questionnaires. To complement this approach, we also conducted focus groups, an online survey, and a lab study. From these user studies, we collected empirical quantitative and qualitative data. Using this data, we derived insights to answer our research questions on ubiquitous notifications. 1.4 Research Context The research for this thesis was conducted between February 2015 and June 2019 at the Institute for Visualization and Interactive Systems at the University of Stuttgart. 1.4.1 Graduate School The graduate school of the SimTech Cluster of Excellence at the University of Stuttgart provided a framework of checkpoints, seminars, and events that supported the research for this thesis and fostered the interdisciplinary exchange with other researchers. Intermediate steps were regularly presented, including a milestone presentation examined by Prof. Dr. Niels Henze and Prof. Dr. Andrea Bart from the Institute of Applied Analysis and Numerical Simulation. 1.4.2 Publications This thesis is based on prior scientific publications [166, 174, 177, 180–185]. The work for these publications was conducted in collaboration with Niels Henze, Albrecht Schmidt, Alexandra Voit, David Hägele, Florian Alt, Frank Bastian, Gisela Kollotzek, Huy Viet Le, Jonas Auda, Lucas van der Vekens, Marcus Hept- ing, Marvin Tiedtke, Philipp Kratzer, Rodrigo Ventura Fierro, Stefan Schneegass, and Sven Mayer. The work led to further publications that are not part of this thesis [172, 176, 178]. 20 1 | Introduction In the following, we present the individual contributions for the parts of this thesis that are based on prior scientific publications: • The work described in Section 3.1 was initiated and driven by the author. He developed and released the system, consisting of an Android app and the server component. He cleaned and analyzed the data set, which he also anonymized and open-sourced. Alexandra Voit and Niels Henze supported the creation of the resulting paper [180], which was published in the proceedings of the conference Mensch und Computer 2019. • Section 3.2 is based on the bachelor thesis project “Annotation and Analysis of Notification Content,” by Gisela Kollotzek in 2018. The author initiated the project and was the primary supervisor of the thesis. Alexandra Voit was the second supervisor, and Niels Henze the examiner. Gisela Kollotzek conducted the case study and developed the first version of the system. The author revised the system and analyzed the collected data for the pa- per [182], which was published in the proceedings of the conference MUM 2019. The paper was driven by the author and supported by Alexandra Voit, Niels Henze, and Gisela Kollotzek. • The system described in Section 4.1 was initiated and developed by the au- thor. The study was conducted and analyzed by the author with the support of Alexandra Voit and Huy Viet Le. Additionally, Niels Henze supported the creation of the resulting workshop paper [185], which was presented in the second iteration of the Smarttention, Please! workshop [131] and published in the adjunct proceedings of the conference MobileHCI 2016. • The work described in Section 4.2 is based on the master thesis project “Investigation of Delay Opportunities of Mobile Notifications,” by Jonas Auda in 2016. The project was initiated by the author, and he was the primary supervisor for the thesis. Alexandra Voit was the second supervisor, and Niels Henze the examiner. Jonas Auda implemented the system and conducted the studies described in the section. For the resulting paper [177], the author analyzed the collected data. The author and Alexandra Voit analyzed the semi-structured interviews. The author drove the paper with the help of all the mentioned parties, as well as Stefan Schneegass. The 1.4 | Research Context 21 paper was published in the proceedings of the conference MobileHCI 2018. Jonas Auda later joined the group led by Stefan Schneegass as a PhD student and drove a follow-up paper [10], which was published in the adjunct proceedings of the conference CHI 2018. • Section 5.1 is based on the bachelor thesis project “Smart Distribution of Notifications Across Multiple Devices,” by Philipp Kratzer in 2015. The author initiated the project and was the supervisor. Niels Henze was the examiner of the thesis. Philipp Kratzer developed the apps described in the section and conducted the study with the help of the author. For the resulting paper [184], the author analyzed the collected data. Alexandra Voit supported the paper writing process. The paper was published in the proceedings of the conference UbiComp 2016. • Section 5.2 is based on the student project “Comparison of the Development of Notification Systems”, by Frank Bastian, David Hägele, and Marvin Tiedtke. The project was initiated by the author (primary supervisor) and Alexandra Voit (second supervisor), and supported by Huy Viet Le (third supervisor). Niels Henze was the examiner of the project. The qualitative user study was conducted by the students and later analyzed by the author and Alexandra Voit. The resulting workshop paper [166] was presented at the third iteration of the UbiTtention workshop [173] and published in the adjunct proceedings of the conference UbiComp 2018. • The work described in Section 6.1 is based on the master thesis project “Notification Mechanisms for Smart TVs,” Rodrigo Ventura Fierro in 2015. The author initiated the project. Niels Henze was the primary supervisor, and the author the second supervisor. Albrecht Schmidt was the examiner of the thesis. Rodrigo Ventura Fierro conducted the studies described in the section with the support of the author and Alexandra Voit. Rodrigo Ventura Fierro developed the first iteration of the lab study system. Based on this, the author developed the system used in the lab study. The author drove the resulting paper [174] with the help of all mentioned parties. Additionally, Sven Mayer helped with the statistical analysis. The paper was published in the proceedings of the conference TVX 2016. 22 1 | Introduction • Section 6.2 is based on the student project “Evaluation of Using Public Displays for Reading Personal Content within Semi-Public Places,” by Gisela Kollotzek, Lucas van der Vekens, and Marcus Hepting in 2017. The project was initiated by the author. The author and Alexandra Voit supervised the project, Albrecht Schmidt was the examiner. Niels Henze and Florian Alt provided additional support and insights. The author outlined the architecture of the system and developed the smartphone app. The students implemented the server and the public display UI and conducted the study. The author and Alexandra Voit analyzed the collected data and interviews. The resulting paper [183] was published in the adjunct proceedings of the conference CHI 2018. • The system described in Chapter 7 was initially created by the author in 2015 and continuously developed since then. In 2018, the author open- sourced the system. Alexandra Voit and Niels Henze provided feedback for the workshop paper [181], which was presented at the third iteration of the UbiTtention workshop [173] and published in the adjunct proceedings of the conference UbiComp 2018. 1.4.3 Funding The following funding bodies partially funded the research for this thesis: SimTech Cluster of Excellence1 Parts of the research were funded by the SimTech Cluster of Excellence at the University of Stuttgart within the project network “Reflexion and Contextualisation” using the working title “Modeling Human Behavior for Smart Notification Management in the Context of Ubiquitous Com- puters.” Design of Adaptive and Ambient Notification Environments (DAAN)2 The re- search for this thesis was partially funded by the German Ministry of Education 1https://www.simtech.uni-stuttgart.de/ 2http://daan.dfki.de/ 1.4 | Research Context 23 https://www.simtech.uni-stuttgart.de/ http://daan.dfki.de/ and Research (BMBF) within the DAAN project [139]. Project partners were the German Research Centre for Artificial Intelligence (DFKI), Deutsche Telekom, IXDS, University of Stuttgart, Intuity Media Lab, and UdK Berlin. SFB-TRR 1611 This work was partially funded by the project “Metrics for Mo- bile Visualization and Interaction Techniques through Research in the Large” within the SFB-TRR 161 “Quantitative Methods for Visual Computing”, sup- ported by the German Research Foundations (DFG). 1.4.4 Collaborations Socio-Cognitive Systems (SCS) Group, University of Stuttgart Within the Socio- Cognitive Systems group at the University of Stuttgart, led by Prof. Dr. Niels Henze, the ongoing collaborations resulted in a number of publications co- authored with Sven Mayer [90] and Alexandra Voit [158, 159, 162, 164–169]. Human-Computer Interaction (HCI) Group, University of Stuttgart The Human- Computer Interaction group at the University of Stuttgart, led by Prof. Dr. Al- brecht Schmidt, closely worked together with the Socio-Cognitive Systems group. The following co-authored publications are the result of collaborations: Alireza Sahami Shirazi et al. [137], Katrin Wolf et al. [192], Lars Lischke et al. [84], Matthias Hoppe et al. [62], Thomas Kubitza et al. [77], Tilman Dingler et al. [34], and Yomna Abdelrahman et al. [1]. External Collaborations Further collaborations include Jonas Auda et al. (Uni- versity Duisburg-Essen) [10] and Frederik Wiehr et al. (DFKI) [190]. Doctoral Colloquia Parts of this work were discussed at two doctoral collo- quia. The first doctoral colloquium was held in conjunction with the conference TVX 2016. It was chaired by Teresa Chambel (University of Lisbon) and Sharon Strover (University of Texas at Austin). Frank Bentley (Verizon Media) sup- ported the discussions. The second doctoral colloquium was held in conjunction with the conference MobileHCI 2017 [172]. It was chaired by Céline Coutrix 1https://www.sfbtrr161.de/ 24 1 | Introduction https://www.sfbtrr161.de/ (Université Grenoble Alpes), Jennifer Pearson (Swansea University), and Andrés Lucero (Aalto University). Mikael B. Skov (Aalborg University) supported the discussions. Workshop Series Parts of this work have been presented at a series of work- shops on smart attention management. The Smarttention, please! workshops in conjunction with the conferences MobileHCI 2015 [131, 176] and 2016 [159, 175, 185]. The UbiTtention workshops in conjunction with the conferences UbiComp 2016 [77, 161, 164, 190], 2017 [105], 2018 [166, 173, 181], and 2019 [40]. The Intelligent Notification and Attention Management on Mobile Devices workshop in conjunction with the conference MUM 2017 [179]. Internships Between October 2016 and February 2017 the work on this thesis was briefly paused due to an internship at Google Research, which was hosted by Yang Li. The work was paused again between May and August 2018 for a second internship at Google Research, which was hosted by Pingmei Xu. 1.5 Thesis Outline This thesis consists of eight chapters, a bibliography, and an appendix. We present the results and evaluations of multiple empirical studies, a review of related work, discussions, and a conclusion. The thesis is structured as follows: Chapter 1 - Introduction Motivates the thesis, defines the research questions, and outlines the research context. Chapter 2 - Background and Related Work Provides background and a review of related work on notifications. Chapter 3 - Notifications on Mobile Devices Describes the assessment of mo- bile notifications, specifically mobile notification drawers and an approach for annotating notifications in user studies. Chapter 4 - Managing Mobile Notifications Introduces two approaches for im- proving the management of mobile notifications: A notification dashboard and the ability to “snooze” notifications. 1.5 | Thesis Outline 25 Chapter 5 - Notifications in Multi-Device Environments Reports a quantita- tive study on notifications in environments with multiple different devices and a qualitative follow-up study. Chapter 6 - Notifications on Large and Pervasive Displays Further expands the set of devices that can notify the user by considering smart TVs and public displays. Chapter 7 - Notification Logging Framework Describes the architecture and use-cases for an open-source notification logging framework for mobile devices that was used throughout this thesis. Chapter 8 - Conclusion and Future Work Summarizes the thesis and the re- search contribution and outlines future work. 26 1 | Introduction 2 Background and Related Work This chapter provides background information and an overview of related work on ubiquitous notifications. The chapter is structured in three parts. We first present notifications and notification systems on current operating systems. We then provide a summary of the author’s prior work that directly inspired the creation of this thesis. Finally, we provide an overview of related work on notifications focusing on mobile notifications. This chapter is meant as an introduction to the topic of ubiquitous notifications. We will complement the related work throughout the following chapters as we expand the focus on more devices. Parts of this chapter are based on the background and related work sections of the publications [166, 174, 177, 180–185] that chapters 3 - 7 are based on. 2.1 Notification Definition The Cambridge Dictionary defines notification as “a message that is automatically sent to you on your mobile phone or computer” [17]. The related term push notification is defined as “a message sent to a smartphone relating to one of its apps, even when it is not running, or the act of sending such messages” [18]. Although the definition explicitly mentions smartphones, all current dominant 27 operating systems for devices like tablets, smartwatches, desktop PCs, and laptops support (push) notifications. In the following, we provide a brief overview of current notification systems on different devices. 2.2 Notification Systems on Different Devices Notifications are not a new phenomenon. Landline phones inform users about incoming calls by ringing. Mobile phones that predate smartphones supported ringtones for incoming calls, SMS, and sometimes reminders. Applications running in the background of desktop computers and laptops informed users about various events, such as system alerts, new emails, and instant messages. However, the scope of notifications mainly was limited to a specific set of events on specific devices. Always connected smartphones that can be easily extended by apps downloaded from app stores removed limitations. Notifications are now an operating system (OS)-level feature that all kinds of apps and services can use for all kinds of events. Other devices followed this approach. Notifications are now a feature on all popular operating systems that application developers can expect to be available, typically through a well-defined application programming interface (API). This thesis focuses on these “modern” notifications and different kinds of smart devices. We will now briefly summarize the current state of notifications on current smart devices. 2.2.1 Smartphones Google’s Android and Apple’s iOS are the current dominant mobile operating systems. Notifications are shown on the lock screen and, therefore, one of the first things users see when turning on the screen of the device (see Figure 2.1a). Apps can trigger notifications, typically represented as rectangular boxes with icons and text, and optionally accompanied by auditory and tactile cues. These notifications are collected in the notification drawer, which can typically be accessed at any time by swiping from the top of the screen (see Figure 2.1b). Users have different levels of control about which apps can trigger notifications and how they are alerted. This ranges from individual settings for specific apps to “do not disturb” modes that affect all notifications. The default interaction with a notification is 28 2 | Background and Related Work (a) Lock screen (b) Notification drawer Figure 2.1: Exemplary smartphone notification on Android 11 shown on the lock screen (left) and in the notification drawer (right). either tapping it or dismissing it by swiping it away. On recent versions of Android and iOS, notifications offer more interaction options. For example, additional buttons trigger other actions or inline replies for instant messages. 2.2.2 Tablets Android and ipadOS (a variant of iOS) are also prominent operating systems for tablets. Overall, notifications are implemented in the same manner as on smartphones. Due to the increased screen size there are slight visual tweaks (see Figure 2.2a), but the overall user experience is very similar to smartphones. One difference is that tablets are often equipped with vibration motors due to their size and notifications can therefore be limited to visual and audio cues only. 2.2 | Notification Systems on Different Devices 29 (a) Tablet (b) Smartwatch Figure 2.2: Exemplary tablet (left) and smartwatch (right) notification. 2.2.3 Smartwartches With Wear OS (previously Android Wear) and watchOS (a variant of iOS), variants of Android and iOS are also represented on smartwatches. Here, notifications are adjusted to fit the smaller form factor (see Figure 2.2b). Interaction is more focused on pre-defined actions that allow users to take action on notifications without requiring them to open app experiences in full screen. There is also a stronger focus on vibrotactile alerts that subtly tap users on the wrist to inform them about new events. A major difference compared to smartphones and tablets, where the source of notifications is typically an app on the device, smartwatches often allow users to “mirror” notifications from a connected device. This allows smartwatches to conserve battery life by only requiring a wireless connection to the smartwatch. All processing of the events and notifications is done on the connected smartwatch. However, recent smartwatches also allow standalone apps on the device itself, to slowly untether the requirement of a connected smartwatch. Another aspect to consider is that, due to the limited form factor of smartwatches, users might be inclined to attend to smartwatch notifications on a different device with a large screen or better-suited input options. 30 2 | Background and Related Work Figure 2.3: Exemplary notification on Windows 10. 2.2.4 Desktop PCs and Laptops The dominant operating systems for desktop PCs and laptops, Microsoft Windows and Apple macOS, adopted OS-level notifications similar to smartphones. This includes a standardized API, similar-looking notifications, and a notification drawer. They also offer similar options to control notifications for individual apps and modes that affect notifications from all apps, such as “Do not Disturb” or “Focus Mode.” The exact controls differ on every platform and even on different OS versions. 2.3 Prior Work by the Author This thesis continues prior research on notifications that the author conducted and was involved in. Although this research is not part of this thesis, it is crucial as a foundation for this thesis. Therefore, we will summarize this work in the following. We will refer to the author of this thesis as Weber throughout the following two sections. 2.3 | Prior Work by the Author 31 2.3.1 Researching Mobile Notifications at Scale In 2014, Sahami Shirazi et al. published the first paper on the large-scale as- sessment of smartphone notifications [137]. During this time, Weber worked as a student assistant, built the system, and collected the data for the paper. The “Desktop Notifications for Android” system was built and an launched at the end of 2012, the data was collected in January - July of 2013, and the paper was published in early 2014. This work contributed the first assessment of notifica- tions at scale, breaking down what kind of notifications users like and dislike, and derived design guidelines for mobile notifications. The data used for the paper consisted of almost 200 million notifications from over 20,000 apps and over 40,000 users. This section summarizes the paper and provides additional background and insights from a perspective of several years in the future. Apart from the direct findings, this work also set an example of how to investigate notifications at scale by giving users a reason to install the study apparatus. The apparatus was a service that allowed users to view their smartphone notifications on their laptop or desktop PC. It consisted of three major parts: an Android app, a browser extension for Mozilla Firefox and Google Chrome, and the back-end server. Users would install the Android app on their smartphones. The app would then generate a secret code and listen for new notifications on the device. Any new notification would then be forwarded to the server and stored in a database along with the code. The users would then enter the secret code in the browser extensions, periodically polling the server with the code to fetch new notifications. If a new notification was found on the server, the browser extension would display it on the user’s desktop. Users were able to exclude notifications from specific apps from being sent to the server. This functionality was included in the first version of the app for two reasons. First, to prevent apps that create many notifications from spamming the user. Second, to give users control of sending notifications with sensitive content to the server and eventually their laptop or desktop PC. Since the notification content was not encrypted on the server, it was accessible by the researchers. However, the analysis was limited to the notification metadata. The system was used to collect multiple sets of data. First, the metadata of the notifications that passed through the server. Second, the list of apps that were 32 2 | Background and Related Work excluded from sending notifications. Third, answers to a questionnaire that was randomly attached to the notifications on the desktop in the form of a button. Finally, the time between creating a notification and the user clicking on it. This led to several interesting findings. For instance, 50% of clicks on notifi- cations happened in the first 30 seconds after the notification was created. Further, the app WhatsApp dominated the list of created notifications. The system also collected 4,964 quantitative ratings and 796 qualitative answers from 4,816 users. This means that the system only collected about one rating per participating user. However, since this was done at scale, it provided insights into how users perceive notifications. The click time was found to be faster for apps that were rated important, and apps that were excluded from sending notifications received lower importance ratings. Finally, notifications from messaging apps notably stood out as more important than other notifications. The researchers concluded the paper with the following five insights about mobile notifications [137]: 1. “[The] nature of notifications is disruptive” 2. “Important notifications do not necessary cause immediate attention” 3. “Notifications are for messaging” 4. “Important notifications are about people and events” 5. “Not all notifications are important” While the system provided novel insights into mobile notifications and users’ preferences and interactions, it was not without limitations. The system simply tried to send out notifications once and then discarded them. If the Android device was offline when a new notification was created, it would not retry to send it. Users also had to dismiss notifications twice. Once on the smartphone and once for the copy on the desktop. This, and the aspect that users might see notifications on the desktop before the smartphone, might also have altered their behavior when interacting with notifications on the smartphone. Rating the notifications for their importance was also done on the desktop instead of the smartphone. Moreover, since there was only about one questionnaire rating per user, there were few insights into generalizable findings, and there was a bias towards apps 2.3 | Prior Work by the Author 33 that created more notifications. Finally, the system required the user to install and set it up on two devices, which might have biased the user base towards more technically savvy users. Although the system was built to research mobile notifications, mirroring notifications on other devices opened new research questions regarding multi- device notifications. 2.3.2 Expanding to Multi-Device Environments In late 2014, Weber extended the “Desktop Notifications for Android” system for his diploma thesis [171]. The update addressed shortcomings of the initial version of the system. It replaced the connection code for pairing devices with a one-click sign-in button that allowed users to sign in with their Google account. All devices signed in with the same account were automatically paired. The system allowed pairing a virtually unlimited number of Android devices and web browsers. Further, the system was updated to broadcast notifications from one device to all other devices, including other Android devices. The system was updated to differentiate between Android smartphones, Android tablets, and web browsers. The simple boolean blacklist was also updated to allow disabling sending notifications to specific types of devices. Users could control whether notifications should be broadcasted to other Android smartphones, Android tablets, and web browsers for each app. Additionally, a new “Private Mode” allowed sending the information that a new notification from a specific app was triggered but omitted sending the actual notification content to the server. For instance, the user would see an email notification on the origin device, including the content of the email, and broadcasted notifications without the content on other devices. Furthermore, dismissing a notification on one device dismissed the notifications on all devices at once. Since users could accidentally dismiss a notification across devices, the update also included a notification history accessible within the Android app. Last, the user could see a list of all devices paired with the account. With this updated system in place, Weber collected new data from over 36,000 apps and more than 33,000 users. A previous limitation was that questionnaires about notifications were triggered on the desktop. This was also changed to show the questionnaires as a notification on Android. Users were asked whether a 34 2 | Background and Related Work specific notification should be shown on a specific device. The options were smart- phone, tablet, smartwatch, smartglasses, desktop PC, laptop, and TV. Smartphone, desktop PC, laptop, and tablet received the highest agreement ratings, followed by smartwatch, smartglasses, and TV. Another questionnaire was attached to the notifications of receiving devices (both Android and in the browser). It showed that “messenger” notifications were valued on every type of device. An analysis of the blacklist and the “Private mode” revealed that apps in the “Tools” category were blacklisted most often. In contrast, communication apps were set to the private mode most often. 2.3.3 Limitations and Learnings While this improved system enabled Weber to collect more data and gain first in- sights into the differences between devices and apps in multi-device environments, there were still several limitations. The app had a large user base, but all changes required a considerable effort to coordinate to implement changes on Android across multiple Android versions, the server, and the browser extensions across multiple browsers and operating systems. The large number of notifications sent through the system every second made it necessary to consider exactly what data to log. Further, handling the actual notification content was still a challenge due to privacy concerns. Finally, user feedback via email and the Google Play Store comments made it clear that users downloaded the system for its functionality and disliked the questionnaires. When shown repeated questionnaires, users clearly expressed their discontent. These limitations made it increasingly difficult to adapt the system to address new research questions. This resulted in several learnings for future work. For example, research probes should be limited to more focused apps to reduce the system complexity. Data collection could be improved by batching log data on-device and periodi- cally sending the data to the server. The data collection should also follow data minimization rules by only collecting data necessary to answer the research ques- tions. Privacy sensitive data should be hashed or encrypted. Finally, depending on the research question, research probes should be deployed in environments with different levels of control. 2.3 | Prior Work by the Author 35 2.4 Related Work In this section, we provide an overview of related work. We first focus on mobile notifications and then expand the scope beyond mobile. We then focus on interruptions caused by notifications and potentially adverse effects. Subsequently, we provide an overview of notification management approaches. Finally, we discuss research in the wild and using app stores for research. 2.4.1 Mobile Notifications Users are confronted with more and more notifications in their daily lives. Notifi- cations are a popular method to engage users and inform them proactively, e.g., about new messages, events, or updates. Notifications use visual, tactile, and auditory cues to gain the users’ attention [66]. 2.4.1.1 Communication and Messaging Nowadays, smartphones, tablets, and an increasing number of wearables have become an essential part of our everyday life. Pielot et al. showed that while communication-related notifications help to make users feel more connected to others, receiving too many notifications can get overwhelming [121]. In an in-situ study with 15 participants, Pielot et al. investigated how users interact with smartphone notifications. Over the course of one week, participants received an average of 63.5 smartphone notifications per day, mostly from instant messaging and email applications. Furthermore, the study showed that notifications are viewed within minutes, even when the smartphone was put in silent mode. In 2018, Pielot et al. revisited mobile notifications in a study with 278 participants [127]. The results again showed the importance of messaging notifications. Participants were fast to attend messaging notifications, while other types of notifications were either removed quickly or left unattended for longer periods. Messaging is a recurrent topic in related work. Instant messaging is a flexible way of communication, that can vary between synchronous and asynchronous discussions [11, 70, 98]. Mobile phones enabled text messaging as a popular communication method [13]. Researchers investigated “traditional” SMS usage and compared it with modern instant messaging (IM) apps such as WhatsApp [23]. 36 2 | Background and Related Work Church et al. found that cost and social influence are reasons for WhatsApp overtaking SMS messaging [23]. Dingler et al. explored the attentiveness of users toward mobile messages [33]. They found that users were attentive to messages for approximately 12 hours per day. Researchers found that participants attend notifications about individual (1:1) chats faster than group chats [127]. Avrahami et al. showed that the responsiveness toward instant messaging is affected by the context and the presentation of messages [11]. Mehrotra et al. found that the sender of messages can have an impact on how notifications are perceived [95] and Pielot et al. identified features to predict if a user will attend a message within a specified period [122]. The number of generated notifications is constantly increasing [121]. Ill- timed notifications can also distract or interrupt the recipient [88, 121]. They can be distracting, might cause negative emotions, or are just not important for their recipients [114, 121, 126, 137]. Czerwinski et al. explored the adverse effects of instant messaging interruptions on different kinds of tasks [30]. Other work looked at the attentional cost of receiving notifications [147] and relevant interruptions [29, 44, 49, 71]. A notification recipient might benefit from valuable information that he or she receives in a proactive manner [30, 95]. 2.4.1.2 Interactive Notifications Notifications are not limited to messaging. Dingler et al. investigated the use of notifications for microlearning sessions on the go [34]. The researchers built an Android app that triggered microlearning notifications at random times and depending on a model [123] that also included the time of the last received notification. 2.4.1.3 Notification Modalities Using visual, tactile, and auditory cues, these devices can use notifications to gain the user’s attention [66]. Hansson et al. compared public with private and subtle with intrusive notification cues [54]. The researchers presented a model for visual- izing the difference between tactile and auditory notifications. According to their model, tactile notifications fall into the subtle and private category, while auditory 2.4 | Related Work 37 notifications are intrusive and public. Exler et al. investigated the perceptibility of different notification types depending on the position of the smartphone [39]. The researchers looked at sound, tactile, and LEDs, with the smartphone being placed on a table, in the pocket, or in the backpack. The results of a lab study with 36 participants showed that overall tactile notifications were favored due to the low obtrusiveness. Sound was favored for important notifications, and using a LED was found to be suitable for unimportant notifications. 2.4.1.4 Logging Notifications A common theme in related work on notification research is the need to log notifications. Pielot et al. [121], Dingler et al. [33], and Mehrotra et al. [94, 95] logged mobile notifications to provide further insights into what kind of notifications users receive on a daily basis and how they are perceived. In a recently published work on the importance of notification content, Visuri et al. were surprised by a participant of a pilot study not clearing their notifications [156]. The researchers asked participants to label notifications regarding how important and timely they were perceived. They found that many users often dismiss or ignore notifications, and the notification content plays an important role in regard to how users act. The researchers suggest applying semantic analysis to detect unwanted notifications, which requires accessing and logging the content of notifications. 2.4.2 Beyond Mobile Notifications are no longer limited to single devices. With smartphones becoming ubiquitous and new kinds of connected devices entering our everyday lives and homes, notifications follow the users throughout the day. For instance, email notifications were once limited to desktop computers. Today, many kinds of devices can alert about incoming emails, including laptops, smartphones, smart- watches, fitness trackers, and tablet computers. In the future, Internet of Things (IoT) devices like smart light bulbs, intelligent speakers, and pervasive displays will also notify the users. All these devices differ in their modalities used to notify users but also in the modalities users can react to notifications. However, 38 2 | Background and Related Work implementation-specific differences determine how users experience these notifi- cations, even for devices of the same type. For instance, in the mobile operating system Android, notifications are designed as opt-out, while on iOS, they are opt-in. 2.4.2.1 Desktop Computers Research on interruptions caused by notifications predates the current set of smart devices. In 2000, Czerwinski et al. investigated the effects of instant messaging on different tasks on desktop computers [30]. Their results show adverse effects of notifications on the task performance. Further, they found that the adverse effects depend on the task type. The authors argued that with the rise in popularity of instant messaging systems, guidelines for minimizing adverse effects and maximizing the value for the users have to be developed. This can lead to adverse effects, such as increased stress [193], inattention [78], and reduced task performance [30]. Czerwinski et al. investigated the effects of interruptions on task switching on traditional desktop PCs [31]. Notifications on desktop computers tend to provide a passive awareness of incoming information rather than prompting users to change their current primary tasks [67]. When notifications are turned off on desktop computers, some users can increase the performance of their primary tasks; however, other users interrupt themselves to check for information manually [67]. While notifications cause interruptions, they are still valued by users because they provide awareness [67]. 2.4.2.2 Smartwatches An investigation of smartwatch usage revealed that smartwatches are used briefly and frequently during the day [153]. Users value that they can quickly check the information on their smartwatches without being considered rude in social interactions and have the opportunity to decide if there is a need to interrupt their current primary tasks. Furthermore, smartwatches offer less disrupting access to incoming notifications than smartphones [19, 128, 153]. Similar to smartphone notifications, users interact more with notifications about communi- cation with other people [136, 153] and calendar events [136] on smartwatches. 2.4 | Related Work 39 Sahami Shirazi and Henze also conducted an in-situ study about notifications on smartwatches [136]. The researchers collected responses in an online survey on smartwatches from 440 participants and contrasted the results with an in-situ study, rating individual notifications on smartwatches. According to the results, the importance of a notification does not only depend on the notification type but also on the device that shows them. Due to the small size of smartwatches, they are more of a “read-only” device and not a replacement for smartphones. Their re- sults further show that, on smartwatches, the most important notification category was not messaging, but calendar and VoIP. Lee et al. provided further insights about notifications on smartwatches [80]. The researcher explored reducing the distraction of smartwatch users with deep learning. Pearson et al. explored a different use case for smartwatches by using them as public displays [115]. The researchers proposed showing different content on smartwatches for the “wearer,” the “glancer,” and the “public.” According to the results of their studies, it is socially acceptable. 2.4.2.3 Ambient Notifications Müller et al. all investigated using ambient light to alert users. The light was positioned in the periphery of the users. In a lab study, they found it to be of similar usefulness as traditional pop-up notifications. However, the authors men- tioned privacy concerns since it can be seen by others. They further discussed that tactile notifications are better suited for private alerts and that this approach still needs to be investigated in situ. Kubitza et al. integrated notifications in an IoT infrastructure for intelligent living environments. [77]. Using this system, the interconnectivity of the IoT devices can be leveraged to, for instance, alert users about a notification on their smartphone using smart light bulbs, intelligent speak- ers, and pervasive displays. Voit et al. developed a smart plant system that notifies users about water levels using ambient light or smartphone notifications [163]. 2.4.2.4 Embodied Notifications Schneegass and Rzayev proposed using electrical muscle stimulation (EMS) for implicit notifications [140]. Instead of explicitly gaining the user’s attention, such 40 2 | Background and Related Work a system could implicitly make the user perform certain actions. For example, making the user twist the arm to look at the watch or move the arm towards the phone. Poguntke (née Kettner) et al. developed a wrist-worn device that applies pressure feedback [74]. According to the researchers, this device can provide tactile notifications with reduced stress levels compared to traditional vibrotactile feedback. With Slappyfications, Günther et al. provided a humorous take on notification research [53]. The researchers proposed using pokes and slaps to notify users. As a final level of escalation, they proposed the STEAM-HAMMER to ensure users cannot miss a notification. 2.4.2.5 Multi-device Environments While a body of work investigated notifications on individual devices, little is known about notifications in multi-device environments. Weber found that there is a need for a mechanism to coordinate the distribution of notifications across the user’s devices [176]. Such a mechanism has to take multiple factors into account, such as when a notification should be optimally delivered and which of the user’s device(s) should display the notification. Regarding when notifications should be optimally delivered, Okoshi et al. developed Attelia II. This system delivers notifications at identified breakpoints based on the user’s multi-device usage and the user’s physical activities [110]. The evaluation results of Attelia II revealed that delivering notifications at breakpoints in multi-device environments reduces the perceived workload of the user. Fallman and Yttergren proposed a system for mobile phones that detects nearby users and chooses an appropriate notification modality accordingly [41]. With NotifyMeHere, Mehrotra et al. explored intelligent notification delivery in multi-device environments [92]. Under the assumption that smartphone notifications are handled on another device if the user did not interact with the smartphone when the notification was dismissed, the authors explored models of whether a user wants to be notified on the smartphone or “another” device. 2.4 | Related Work 41 2.4.3 Interruptions and Adverse Effects While notifications allow us to be connected, they can also cause interruptions. The disruptive nature of interruptions and task switching has been an important research topic for many years [31, 63]. While not all interruptions are disrup- tive [47], Adamczyk and Bailey showed that different timings of interruptions have different effects on users [2]. Mehrotra et al. found that the perceived disrup- tion of a notification is influenced by several factors, including the notification’s presentation, the relationship between the sender and receiver, and the task the user is engaged in [94, 95]. Prior studies investigated what makes interruptions disruptive [47]. Interruptions can delay task completion by up to four times [83]. While interruptions may cause inattention [78], intense phone use does not predict negative well-being [72]. In a world of constant connection, being unavailable is an interesting research topic [15]. Aranda and Baig discussed how users are more and more dependent on smartphones, difficulty to disconnect, and “the fear of missing out” [9]. Mehrotra et al. investigated the effect of cognitive and physical factors on the response time and the disruption caused by interruptions through incoming notifications [95]. In terms of negative effects, work by Leiva et al. shows that interruptions caused by mobile notifications introduce a significant overhead when completing tasks [83]. Recent work by Kushlev et al. shows that smartphone notifications increase inattention and hyperactivity symptoms [78]. Smartphone users often do not realize how many notifications they receive [185]. Sahami et al. found that a large number of notifications are issued by messaging applications [137, 176]. On the one hand, users value notifications issued by such applications. On the other hand, not all notifications that users receive are considered important. Church and Oliveira compared SMS to instant messaging applications like WhatsApp [23]. Their study revealed several concerns regarding WhatsApp messages and notifications, e.g., coping with too many messages or interruptions and the fear of missing business-related messages if notification modalities are switched off. Understanding how users handle messaging noti- fications might help to build messaging services which do not overload users by issuing too many notifications. Further negative effects include decreased productivity and slower and more error-prone performance [2, 12, 49, 117, 138]. 42 2 | Background and Related Work Exler et al. surveyed 68 participants on how smartphone notifications are interrupting and disturbing at specific locations [36]. They found that users are more receptive to interruptions while waiting (e.g., bus stations and parking lots) and less at movie theaters, libraries, and restaurants. Mayer et al. evaluated how mobile notifications are disrupting conversations [89]. The researchers set up a simulated conversation environment, and used eye tracking combined with a qualitative analysis. Simply disabling notifications entirely is no suitable solution [125, 126]. Thus, managing notifications to not continuously disturb the user is a crucial task. The type of the primary task, its complexity, its duration, the length and number of interruptions influence the perceived difficulty of continuing a task after an interruption [31]. In a diary study, Czerwinski et al. showed that returning to tasks after being interrupted is hard [31]. Vardhan et al. discussed the balance of convenience and privacy of mobile notifications [152], and Lee et al. investigated smartphone “overuse” and the role of messaging [82]. 2.4.4 Notification Management A body of prior work has explored how mobile notifications can be better managed. Researchers investigated what users do when they sense notifications [21] and which strategies users apply to cope with notifications. Gallud and Tesoriero suggest moving from sound to visual notifications [45]. Auda et al. explored a system for rule-based notification deferral by suppressing, summarizing, or automatically snoozing notifications [10]. Mehrotra et al. took this a step further by automatically suggesting rules based on usage patterns [93]. The researchers found that the notification’s title and the user’s location can be used as features to determine whether a message will be dismissed. 2.4.4.1 Call Predictions Phone calls are urgent notifications that users have to attend in a small timeframe in order to not miss the call [118]. Using anonymous data from 418 users, Pielot created a model to predict whether a user will pick up a call. The researcher was 2.4 | Related Work 43 able to predict this with an accuracy of 83.2% by using features such as when the user was last using the device, the time passed since the last call, when the ringer mode was changed last, and the device orientation. 2.4.4.2 Opportune Moments and Breakpoints A number of research projects are focusing on the approach to delivering no- tifications at opportune moments instead of delivering them immediately [43, 44, 114, 130, 143]. With Attelia, Okoshi et al. developed a middleware that defers notifications to so-called breakpoints - times between two consecutive activities [106, 109–111]. Deferring notifications to these breakpoints has been shown to lessen disruptive effects [44]; however, this has to be balanced with social expectations to reply quickly [177]. Attelia runs on the user’s smartphone and can detect breakpoints of the user’s activity on his or her mobile device. Further, it can detect physical breakpoints through smartwatches. According to the researchers, determining which device to notify the user on is a challenge for future research. Okoshi et al. conducted an in-lab and an in-the-wild study of Attelia [108]. The results showed significantly reduced frustration if interruptions were triggered during breakpoints. Okoshi et al. had the opportunity to conduct a real-world, large-scale study within the Yahoo! Japan app [112, 113]. More than 680,000 users participated in this study, with the goal of detecting opportune moments to interrupt users. The researchers found a significant reduction of response time compared to issuing notifications directly. Their model initially performed worse on weekends, but due to the large amount of data collected, they were able to improve the model quickly. Tsubouchi and Okoshi followed up on their large-scale research in the Yahoo! Japan app for detecting interruptibility based on activity [149]. The researchers tweaked the features of the model. Thanks to the large amount of data collected, it was possible to adapt and improve the model quickly. Other approaches explored models to better time interruptions [2, 116, 148, 150]. SCAN is another approach of a notification system that takes the social context into account [114]. Fischer et al. investigated mobile phone activity as an indicator of opportune moments to deliver notifications [43]. Iqbal and Bailey investigated the effects of intelligent notification management on users and their 44 2 | Background and Related Work tasks [65]. The researchers built a system that uses statistical models to defer notifications until breakpoints, resulting in reduced frustration and reaction time. Using a context-aware computing device, Ho and Intille detected activity transi- tions [60]. They found that messages delivered in this activity transitions were better received. Using machine learning techniques, Pielot et al. investigated the possibility of predicting the user’s attentiveness to text messages [122]. Mehrotra et al. also presented a system that generates notification rules based on received notifications [93]. To voluntary engage users to interact with recommended content, Pielot et al. used a machine learning approach to determine opportune moments for notification delivery [120]. Poguntke et al. investigated different delay modes for notifications [129]. They compared a fixed interval of one hour, a user-defined interval (defaulted to 10 minutes), and a sender-dependent interval (defaulted to an hour). Anderson et al. recently published a survey on attention management systems [5]. 2.4.4.3 Sensing Context Pielot et al. explored boredom detection and using the boredom state to send out proactive recommendations [119, 123]. They proposed sending out fewer recommendations when the user is busy and more when the user is bored. Dingler et al. investigated if detected boredom can be used to engage a user in micro- learning sessions through notifications [34]. Goyal and Fussell explored timing interruptions based on electrodermal ac- tivity derived from galvanic skin response [51]. According to their results, this approach resulted in significantly reduced distractions. Exler et al. investigated the detection of a smartphone user’s distraction based on typing and touch ges- tures [38]. The results of their study showed that users typed slower and made more errors depending on the workload. The researchers conclude that this in- sight can be used as a measure of distraction. Visuri and van Berkel published a survey paper on the importance of attention in human-computer interaction and an overview of mobile sensing [154]. 2.4 | Related Work 45 2.4.4.4 People and Content The importance and urgency of notifications depend on their content and con- text [94]. Mehrotra et al. used the context of a notification recipient in combination with the content of the notification to realize a non-disruptive notification mecha- nism [94]. In the case of communication-related notifications, the relationship between the sender and the user matters as well [95]. Users might not accept a notification management system that removes important notifications [95]. 2.4.4.5 Research Tools Pejovic and Musolesi proposed InterruptMe, a library for interruption manage- ment for Android [116]. Obuchi et al. investigated pushing ESM questionnaires when breakpoints in a user’s activity are detected [104]. The authors report up to a 70% improvement in the response time when the user’s activity switched from “walking” to being “stationary.” Okoshi et al. proposed creating an “Interruptibil- ity Layer” as a middleware on top of the operating system [107]. The researchers highlighted that this likely would need to be developed in collaboration with the creators of the dominant operating system developers, such as Apple (iOS) and Google (Android). PrefMiner is a system to generate rules for notification management automat- ically [93]. In his PhD thesis, Mehrotra proposed a framework for intelligent mobile notifications [91]. He explored multiple models to predict opportune mo- ments for notification delivery. Further, Mehrotra et al. investigated how mobile experience sampling can be improved [96]. While mobile experience sampling is a useful source of data, the quality of data the varies. One reason for this is that users might be too busy to attend the experience sampling prompts. The researchers suggested detecting breakpoints for opportune moments to prompt users for questionnaires. Visuri et al. proposed a cluster-based user model for predicting interruptibility for manual data collection [155]. A use case for this is quantified-self applications, which trigger alert dialogs for data collection. 46 2 | Background and Related Work 2.4.5 Research in the Wild A challenge of research on mobile notifications is that they are highly context- dependent and received around the clock. To overcome this challenge, some prior work in this area moved from lab studies to in-the-wild studies. A series of publications were concerned with whether it is “worth the hassle” to conduct research in the field. The authors found that at the time (2004), most HCI projects conducted their evaluations in the lab [76]. The authors argue that while “mobile systems are highly context-dependent,” conducting studies in the field is “difficult,” “time consuming,” and the “added value is unknown.” The authors tested a system for usability problems in a lab in a field condition in their work. They found similar usability problems in both conditions but argued that there is a “lack of control” in the field condition. On the other hand, they argue that it is challenging to ensure that everything is covered in the lab condition. For this particular system and study, the authors concluded that both lab and field studies have advantages and disadvantages. Since they found similar usability problems in both conditions, they conclude that the added value of conducting studies in the field is “very little.” In a follow-up work in 2006, a different set of authors (Nielsen et al.) also compared lab and field environments in empirical studies [102]. In their compari- son, they identified significantly more usability problems in the field. The authors explain that the field condition revealed problems with interactions and cognitive load that was not identified in the lab and concluded that it is indeed “worth the hassle” to conduct research in the field. A year later, in 2007, Rogers et al. investigated why it is worth the has- sle [133]. The authors argue that evaluating applications in ubiquitous computing environments is challenging due to their context of use. They explain that metrics used in traditional studies in a lab are optimized for this controlled environment, thus, failing to capture all aspects in more uncontrolled environments. The au- thors further mention living labs that are designed to simulate real environments to counter these effects. In the paper, the authors discuss the questions of how long should studies in ubiquitous computing environments be conducted, how much and what data to collect, and how the findings can be fed back into the design process. They discuss the challenges of evaluating ubiquitous comput- 2.4 | Related Work 47 ing applications that are used over extended periods while users are on the go and doing other things. They mention adapting existing metrics and heuristics and using new intervention evaluation methods such as the experience sampling method. The authors describe case studies for in-situ research and conclude that this effort was successful but expensive in both time and the required effort. The results of the case studies show that here the lab was no option, as the context was needed to capture all aspects and that the environment directly affects the user experience. The authors conclude the paper with the following statement: “Finally, it is impossible, and nor is it desirable, to capture everything when in situ. The key is to use various methods that reveal both hoped for and unexpected effects of the context of use.” [133] In 2014, ten years after the first paper asking whether conducting studies in the field is worth the hassle, two of the original authors looked at the state of mobile HCI research in the past decade [75]. Kjeldskov and Skov conducted a literature review regarding field and lab studies in the mobile HCI context. They conclude that in the end, both approaches are needed. They summarize that in lab studies “data is typically gathered with precise instruments [...] in an artificial environment where it cannot be disturbed from the outside.” According to the authors, the advantages of lab studies are the “ability to focus on detail,” “high replicability,” and “large experimental control.” The disadvantages are “limited relations to the real world,” “unknown external validity,” and “typically low level of ecological validity.” They found that in field studies, data is usually “gathered through observations, interviews and surveying techniques.” The advantages are capturing a “large amount of rich and grounded data” with a “high level of ecological validity.” The disadvantages are “unknown biases” in the field, “unknown external validity/generalizability,” and “typically low level of control.” While both approaches are needed, the authors highlighted that with mobile HCI evolving, the need to consider the complexity of the world is increasing. The authors go further to suggest that the uncontrollable nature of field studies should be embraced. They argue that the value of field studies is that they are “real” and “messy.” Further, they highlight the opportunity to conduct field studies over 48 2 | Background and Related Work extended periods to capture the sustained use of systems. The authors conclude their paper with the sentiment that conducting studies in the lab or the field should not be a matter of “if or why” but rather a “when and how.” 2.4.6 App Stores and External Validity Henze and Pielot explored how app stores can provide external validity for mobile HCI [56]. In the article, the researchers discuss that “considering realistic contexts in traditional lab studies is often not even possible because we know too little about what the realistic contexts are” [56]. They further provide examples of research probes that leveraged the reach of app stores to distribute applications to users in the wild. Henze et al. also discussed the trade-off between opt-in and opt-out for consent in in-the-wild studies [57]. While opt-out allows for greater data collection, it also poses legal and ethical challenges. Using multiple case studies, the researchers found that many users may use apps only for short periods and that users expect research apps to offer a similar user experience to commercial products. An example of such a research app was the Desktop Notifications service that allowed users to synchronize notifications across devices while enabling researchers to gain insights on notifications in-the-wild from a large user base [176]. In 2013, Henze et al. published “ten steps to conduct a large-scale study” [58]. The steps are as follows [58]: (1) Identify the research goals. (2) Select a study method and (3) devise an incentive mechanism. (4) Then select the target platform, and (5) develop the application, (6) including a mechanism to collect data. (7) The app should provide informed consent about what data is being collected. (8) Then publish the application, and (9) continuously monitor the data. (10) Finally, filter and analyze the data to answer the research questions. Exler et al. discuss the difficulties of creating data sets [37]. In particular, they investigated using community-driven data sets using crowd-funded data. The authors discuss issues and limitations, such as data labeling. 2.4 | Related Work 49 2.5 Summary In this chapter, we provided a brief overview of how notifications on different kind of devices are currently implemented. We then provided a summary of the author’s prior work that directly precedes the work in this thesis. Subsequently, we expanded to scope to provide an overview of related work. We first discussed mobile notifications on ubiquitous smartphones. Then we then looked at work beyond smartphone notifications. A common theme in notification research is in- terruptions and adverse effects caused by notifications. We provided an overview of work in this field and followed the section by discussing notification manage- ment approaches to reduce these adverse effects. Subsequently, we discussed the different approaches of notification research in the lab compared to in-the-wild studies. Finally, we briefly provided an overview of how to conduct studies with a high external validity by leveraging app stores. Prior work has shown how many smartphone notifications users receive per day. However, what is not yet known is how these notifications materialize on smartphones and how users manage them (RQ1). Another aspect of notifications is the perceived importance which depends on the notification content. Research- ing notification content is challenging, as this raises privacy concerns. Another open question is assessing notification content in detail while respecting users’ privacy (RQ2). While a number of approaches to improve notification manage- ment exist, this is not a solved problem yet. The next question is, therefore, how we can support users with managing notifications (RQ3). Looking beyond the smartphone, we have seen that research started to expand to other devices. However, most research focuses on single devices at a time. An open question is how various types of personal devices differ in multi-device environments regarding displaying notifications (RQ4). Nowadays, smart TVs are a common type of device in multi-device environments. There is little research regarding the considerations when displaying notifications on smart TVs (RQ5). Finally, expanding the scope further, public displays are becoming more and more ubiqui- tous. The final research question concerns the considerations when displaying notifications on public displays (RQ6). In the following chapters, we will address these research questions. 50 2 | Background and Related Work 3 Notifications on Mobile Devices In the past decade, smartphones exploded in popularity. They are always con- nected and always with the user. Coupled with increased processing power and high-resolution touch screens, this created a paradigm shift in how we consume information. While traditional mobile phones are mostly limited to specific fea- tures, such as phone calls, text messages, and alarms, smartphones can be easily extended by downloading additional apps from app stores. And with notifications being a core feature of smartphones, these apps can proactively provide users with information from a multitude of services. From new email notifications to breaking news and social media updates, in many cases users do not need to open apps to receive new information. Prior work already investigated which kind of notifications users receive on their smartphone [121], how fast they attend notifications [137], and the effect of interruptions caused by notifications [31, 63]. However, what is still missing is an understanding about how notifications materialize on smartphones. Current smart- phones show notifications in notification drawers until they are either attended on or dismissed. How many notifications users let accumulate in notification drawers and whether there are different strategies for managing these notifications are still open research questions (RQ1). Further, a major challenge when researching 51 notifications is that they are inherently personal. Prior work focused on reporting aggregated information about notifications to provide external validity but also to protect participants’ privacy. Another important research question is how we can enable researchers to gain deeper insights into notifications, i.e., the actual content of notifications, while protecting the participants’ privacy (RQ2). In this chapter, we will first report the results of a large-scale observational in-the-wild study on mobile notification drawers. In the second part of the chapter, we introduce a privacy-aware system for annotating notifications in user studies. Parts of this chapter are based on the following publications: D. Weber, A. Voit, and N. Henze. “Clear All: A Large-Scale Observational Study on Mobile Notification Drawers.” In: Proceedings of Mensch und Computer 2019. MuC ’19. Hamburg, Germany: ACM, 2019, pp. 361–372. ISBN: 978-1-4503-7198-8. DOI: 10.1145/3340764.3340765 D. Weber, A. Voit, G. Kollotzek, and N. Henze. “Annotif: A System for Annotating Mobile Notifcations in User Studies.” In: Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia. MUM ’19. Pisa, Italy: ACM, 2019, 24:1–24:12. ISBN: 978-1-4503-7624-2. DOI: 10.1145/3365610.3365611 3.1 Mobile Notification Drawers When users of current smartphones turn on the display of the device, they are usually greeted with a lock screen, consisting of a large clock and a list of notifications below it. On the dominant mobile operating systems Android and iOS, this list of notifications can also be accessed at any time by swiping down from the top of the screen. This universally accessible list is an important feature of current smartphones, as it enables asynchronous communication and provides users with proactive information. The notification list is commonly referred to as the notification drawer (Android), notification center (iOS), notification tray, or notification panel. We use the term notification drawer throughout this thesis. Although a large body of prior work on notifications exists, the notification drawer on smartphones as the central place to view and attend notifications has not been explored in detail so far. However, this is a crucial aspect for a complete un- 52 3 | Notifications on Mobile Devices https://doi.org/10.1145/3340764.3340765 https://doi.org/10.1145/3365610.3365611 derstanding of mobile notifications. In the following, we complement prior work by reporting the results of a large-scale observational study on notification drawers of current smartphones. Using a research-in-the-wild approach, we periodically sampled the contents of notification drawers on Android devices. We collected 8.8 million notification drawer snapshots from almost four thousand devices. Based on this data, we present a novel analysis on the number of notifications in notifi- cation drawers and the positioning of different notification categories. Further, we propose three different user types regarding the management of notifications. 3.1.1 Notifications on Android Notifications were an integral feature of the Android mobile operating system since the first version. Notifications are opt-out, meaning that all installed apps can post notifications by default without asking the user for permission. Notifications may use visual, tactile, or sound cues to gain the user’s attention [66]. All notifications end up in the notification drawer that is accessible by swiping down from the top of the screen (see Figure 3.1). Since Android 5.0, notifications are shown on the lock screen by default as well. Notifications can contain action buttons [34], expandable text, and images. Users can click on notifications to take action or swipe to the left or right to dismiss them. By clicking “clear all,” users can dismiss all notifications at once. As summarized in Chapter 2, a body of prior work investigated which noti- fications users receive, how they are valued, interruptions, and means to reduce adverse effects. However, the notification drawer as the central place to view and attend notifications has yet to be investigated. To fill this gap in prior work and to create a more complete understanding of mobile notifications, we explored notifications drawers in an in-the-wild study. 3.1.2 Study We conducted a large-scale observational in-the-wild study on the content of notification drawers. Our research question was how many and which kind of no- tifications can be found in notification drawers, and whether different notification management approaches exist. 3.1 | Mobile Notification Drawers 53 Figure 3.1: The Android 9.0 (Pie) notification drawer showing four different kinds of notifications about a new message, news articles, traffic updates, and the current weather. 3.1.2.1 Apparatus We developed an Android app that allowed us to snapshot the content of noti- fication drawers in-the-wild in an unobtrusive manner. Our goal was for users to install the app on their own, without explicitly recruiting participants. We developed an Android app that allows users to log and explore their notifications 54 3 | Notifications on Mobile Devices in a local history. The added value for users is the option to look up accidentally dismissed notifications or reflect on notifications they received throughout the day. The app supports the Android versions 5.0 - 9.0, with Android 9.0 being the most recent Android version when this work was conducted. According to the Android distribution dashboard [6], this covered 96.50% of all active Android devices. The app uses the Notification Listener Service API [7] to access the notifications on the device. The notification access for the app has to be explicitly enabled by the user in the device settings. Once enabled, all newly created notifications are stored in a local SQLite database. Users can then browse their notifications in a list and select individual notifications to read the text in detail. 3.1.2.2 Data Collection and Consent After the user installed the app and permitted the app to access the device’s notifications, the app displayed a dialog asking the user to opt into the anonymous data collection. Inspired by prior research on asking for consent in in-the-wild studies [124], the dialog contained the options “Agree” and “No thanks.” The dialog was only shown once. Users could also enable or disable the data collection at any later point in time in the app’s settings. Declining the anonymous data collection did not negatively impact the main functionality of the app in any way. If the user consented to the data collection, the app would periodically snapshot all pending notifications in the notification drawer. We used the Android- Job library [35] to schedule the sampling. The library abstracts from version differences in the Android SDK. We set the sampling job to be executed every 15 minutes, which is the minimum amount of time between two jobs. In later versions of Android, these jobs might be deferred if the device uses battery-saving features such as the Doze Mode, which defers background processes if the device was not used and not moved for a certain amount of time. Each snapshot contained the following features: • A randomly generated unique ID for the device (UUID) to associate multi- ple snapshots with a specific device. 3.1 | Mobile Notification Drawers 55 • The current Android version, device model, product name, and device manufacturer. • The current timestamp and timezone. • Metadata of all notifications in the notification drawer, such as the package name, timestamp of creation and position in the drawer. Snapshots generated by the app were limited to metadata and did not contain text or images. The snapshots were stored in a separate local SQLite database. 3.1.2.3 Procedure We published the study app on the Google Play Store. Users from all over the world were able to download it for free. We did not advertise the app in any way. Instead, users found the app using the Google Play Store search or by reading articles and watching vi