Visualization Research Center (VISUS) University of Stuttgart Allmandring 19 D–70569 Stuttgart Masterarbeit Designing the Next Smart Chair based on a Posture Recognition and Feedback Literature Review Christian Krauter Course of Study: Informatik Examiner: Prof. Dr. Michael Sedlmair Supervisor: Katrin Angerbauer, M.Sc. Aimée Sousa Calepso, M.Sc. Alexander Achberger, M.Sc. Jun.-Prof. Dr. Sven Mayer Commenced: June 22, 2022 Completed: December 22, 2022 Kurzfassung Sitzen gilt als ungesund und es gibt einen anhaltenden Trend zu immer längerem Sitzen, ohne dass ein Ende dieser Entwicklung absehbar ist. Die Forschung legt nahe, dass wir häufige Sitzpausen einlegen und nicht über längere Zeiträume in derselben Haltung sitzen sollten. Pausen können bereits von unseren intelligenten Geräten vorgeschlagen werden, aber für ein differenzierteres Feedback darüber, wie wir sitzen, ist spezielle Hard- und Software erforderlich, zwei Themen zu denen bereits viel geforscht wurde. Diese Arbeit leistet folgende Beiträge zur aktuellen Forschung: Erstens wurde eine umfassende Literaturrecherche durchgeführt. In dieser betrachten wir Hardware zur Erkennung der Sitzhaltung und Feedback, dass den Nutzern helfen soll, sich bewusst zu machen, wie sie sitzen und wie sie dies gesünder tun könnten. Darüber hinaus haben wir einen Prototyp eines intelligenten Stuhls gebaut, der das Sitzverhalten des Benutzers mit Sensoren erfasst und darauf basierend visuelles Feedback gibt. Unsere Literaturrecherche zeigt, dass das Erkennen von Sitzverhalten und das Bereitstellen von Feedback dazu auf unterschiedliche Weisen gelöst werden kann. Unser Ziel war es, einen Überblick über das Forschungsgebiet zu geben, wobei wir einen besonderen Schwerpunkt auf die Drucksensor- Hardware und auf visuelles Feedback gelegt haben. Basierend darauf schlagen wir vor, bei der Auswahl der Techniken folgende Faktoren in Betracht zu ziehen: die Umgebung, die Bedenken hinsichtlich der Privatsphäre; sowie die Kosten, die Portabilität, und Genauigkeit der Hardware. Des Weiteren sollten die Fähigkeiten und die Vorlieben der Nutzer berücksichtigt werden. Wir stellten auch fest, dass ein Bedarf an weiteren und umfassenderen Nutzerstudien besteht, welche die Auswirkungen verschiedener Arten von visuellem Feedback untersuchen. Unser Prototyp eines intelligenten Stuhls ist mit vier Drucksensoren in der Sitzfläche und drei Sensoren zur Entfernungsmessung in der Rückenlehne ausgestattet. Wir halten unseren Prototyp für einen ersten Ausgangspunkt für weitere Arbeiten, die unseren Ansatz erweitern wollen. Eine mögliche Erweiterung wäre das Hinzufügen eines Algorithmus zur Klassifizierung der Sitzhaltung. Es könnten auch andere Feedback-Varianten hinzugefügt und durch Nutzerstudien evaluiert werden, um mehr Erkenntnisse über verschiedene Methoden und ihre Auswirkungen auf das Sitzverhalten zu gewinnen. Darüber hinaus schlagen wir vor, unsere Literaturrecherche zu erweitern, um ein breiteres Verständnis über das Erkennen von Sitzverhalten und das Bereitstellen von Feedback dazu zu erhalten. 3 Abstract Sitting is considered unhealthy, and there is an ongoing trend towards more and longer time spent seated, with no foreseeable end to this development. Research indicates that we should take frequent breaks from sitting and not sit in the same posture for extended periods. Our smart gadgets can already suggest breaks, but for more nuanced feedback about how we sit, special hardware and software are needed, two topics on which much research has already been done. Our contribution to this research is two-fold: First, we conducted a literature review of hardware used to recognize sitting posture and of feedback to help the users be aware of how they sit and suggest ways to do so healthier. Second, we built a smart chair prototype that uses various sensors to measure how the user sits and then provides visual feedback about that. Our literature review shows that recognizing sitting behavior and giving feedback about it can be solved in different ways. Our aim was to give a broad overview of the research area while putting additional focus on pressure-sensing hardware and visual feedback. Based on this, we propose to consider the following factors when selecting techniques: the environment, cost, privacy concerns, portability, and accuracy. Furthermore, the user’s capabilities and preferences should be taken into account. We also found a need for additional and more comprehensive user studies that examine the effects of different types of visual feedback. Our smart chair prototype is equipped with four pressure sensors in the seat and three distance-measuring sensors in the backrest. We consider our prototype to be an initial starting point for further work that aims to extend our approach. One possible extension would be to add a sitting posture classification algorithm. Other feedback variants could also be added and evaluated through user studies to gain more knowledge about different methods and their effects on sitting behavior. Furthermore, we suggest future work to expand upon our literature review to get a broader understanding of sitting posture recognition and feedback. 5 Contents 1 Introduction 15 2 Related Work 17 2.1 Sitting and Its Effects on Health . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2 Mitigation Strategies and Countermeasures . . . . . . . . . . . . . . . . . . . . 18 3 Literature Review about Sitting Posture Recognition and Feedback 21 3.1 Sitting Posture Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.1.1 Wearable, Optical, and Rarely Used Setups . . . . . . . . . . . . . . . . 21 3.1.2 Pressure Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.1.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.1.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2 Sitting Posture Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2.1 Active Correction, Aural, and Vibrotactile Feedback . . . . . . . . . . . 26 3.2.2 Visual Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.2.2.1 Summaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.2.2.2 Physical Objects . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.2.2.3 Non-physical and Real-Time . . . . . . . . . . . . . . . . . . 29 3.2.2.4 Other Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.2.2.5 User Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.2.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.3 Summary and Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4 Smart Chair Prototype Design 41 4.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 4.2 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.2.1 Data Collection on the Microcontroller . . . . . . . . . . . . . . . . . . 44 4.2.2 Python Module for Data Handling and Feedback . . . . . . . . . . . . . 44 4.2.2.1 Data Conversion and Storage . . . . . . . . . . . . . . . . . . 45 4.2.2.2 Visual Feedback . . . . . . . . . . . . . . . . . . . . . . . . . 46 5 Discussion and Limitations 49 5.1 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 5.2 Smart Chair Prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 6 Conclusion 53 Bibliography 55 A Appendix 79 7 List of Figures 3.1 Examples of wearable sensors for sitting posture recognition: RFID tags and a gyroscope. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.2 Examples of a Hard-PS load cell and a Soft-PS textile sensor. . . . . . . . . . . 23 3.3 Examples of actively correcting the user’s sitting posture: inflatable bladders, self-adjusting computer display, and self-adjusting chair and desk. . . . . . . . . 26 3.4 Examples of visual feedback summarizing information about the user’s sitting posture: using a bar chart for time spent in different postures and pie charts for posture balance, showing time spent in different postures in a dial chart, and visualizing multiple sitting posture parameters as a rounded star plot. . . . . . . . 28 3.5 Examples of visual feedback using physical objects: attaching LEDs to a chair or a shirt and creating a physical flower that mirrors the user’s posture. . . . . . . . . 29 3.6 Examples of real-time visual feedback: a smart mirror with body parts highlighted that deviate from good posture, a sketched chair with a heatmap of the pressure, and a desktop notification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.7 Examples of real-time visual feedback: an anthropomorphized giraffe with infor- mation and suggestions about the user’s sitting posture and an anthropomorphized flower enhanced through gamification. . . . . . . . . . . . . . . . . . . . . . . . 31 4.1 Our smart chair prototype shown from the front, where the Time of Flight (ToF) sensors can be seen, and from the side with the enclosure of the microcontroller and the power bank visible in the lower right corner. . . . . . . . . . . . . . . . . 42 4.2 Examples of inspiration for our sensor placement and 3D printed plates to increase the area of the pressure sensors. . . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.3 The sensors we used for our prototype: one of three Time of Flight (ToF) sensors and one of four Force Sensitive Resistors (FSRs) attached to an electronics board. 44 4.4 Images of the hardware components attached to the chair: The Force Sensitive Resistors (FSRs) on the chair’s seat with two of the four 3D-printed circular cones that increase their weight-detecting surface area. The laser-based Time of Flight (ToF) sensors screwed to the chair’s backrest from the back. And the ESP32 microcontroller in an enclosure for protection and easy attachment with a connected power bank. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.5 The wiring between the sensors and the ESP32 microcontroller as a simplified circuit diagram and as an image where the installation of the microcontroller into its enclosure can be seen as well. . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.6 Screenshots of our SketchVis feedback with a sketch of our chair from the side and top. A stick figure represents the user sitting on the chair, while colored circles represent the pressure values. Four different sitting postures are represented: sitting straight and even, hunched forward, leaned back, and not sitting on the chair. . . . 47 4.7 A screenshot of our python program’s main window with the displayed sensor values. 47 9 A.1 Technical sketch showing our smart chair prototype’s seat from the front and above, where the Force Sensitive Resistors (FSRs) are represented by green circles. . . . 81 A.2 Technical sketch showing our smart chair prototype’s backrest from the front where the Time of Flight (ToF) sensors are represented by orange circles. . . . . . . . . 82 10 List of Tables 3.1 Overview of publications containing visual feedback for sitting postures. . . . . . 33 3.2 (Cont.) Overview of publications containing visual feedback for sitting postures. 34 3.3 The number of times different visual types were for sitting posture feedback by the publications listed in Tables 3.1 and 3.2. . . . . . . . . . . . . . . . . . . . . . . 35 A.1 Overview of publications containing non-visual sitting posture feedback. . . . . . 79 A.2 (Cont.) Overview of publications containing non-visual sitting posture feedback. 80 11 Acronyms ADC Analog to Digital Converter. 43, 44, 51 CLI Command Line Interface. 44–46 FSR Force Sensitive Resistor. 23–25, 41–47, 50, 51 IMU Inertial Measurement Unit. 21, 24 SB Sedentary Behavior. 17–19 ToF Time of Flight. 41–47, 50 13 1 Introduction A lot of time in our lives is spent sitting. We sit during a large part of our commutes to school, university, or work, places where most of us continue to sit for a substantial part of the day. Back at home, we also sit a lot while watching TV, movies, and series or browsing social media. How long we are sitting on average is difficult to evaluate, but multiple studies in the literature found the average to be around five hours per day [1–3], while it can go up to 12.5 hours per day for some work sectors [4]. However, the general trend of increased sitting time is evident in long-term data, as can be seen for the EU between 2002 and 2017 [5], where the number of people sitting for more than 4.5 hours a day expanded from 49.3% to 54.5%. This trend got amplified during the COVID-19 pandemic because of the increase in working from home [6] and the imposed restrictions on public life, especially lockdowns [7, 8]. The high amount of time we spend sitting and the increase thereof is concerning, as many studies have shown that prolonged sitting is detrimental to our health. It can be the cause of exhaustion, musculoskeletal disorder symptoms, and even premature mortality [9–14]. To mitigate these effects, one could break up sitting time with physical activity [9, 15–24] or by standing up [25–27]. However, there is no clear consensus in the literature on the optimal frequency and intensity of such breaks [21]. Besides reducing the amount of sitting time, one can also focus on improving the quality of sitting, namely the sitting posture. Although there are general guidelines [28] and sitting postures commonly viewed as better than others [10, 29–31], recent research suggests that the importance lies in the frequent change of sitting postures [21, 28]. Regularly changing our sitting posture or getting up can be difficult, though, especially if we are engaged in our current task. The straightforward idea for a solution to this problem is the use of feedback that reminds us of how long we have not been moving. Three things are required to provide such reminders in any form: 1) a way to measure how someone is sitting, 2) software that analyses the data, and 3) an approach to inform the user about the current state and suggest a change. A large amount of research has been conducted on various technologies and techniques for detecting and classifying sitting postures [32, 33]. Giving someone feedback about their posture has also been studied extensively, exploring various modalities such as vibration, sound, visualizations, and hardware that actively corrects the user’s posture. To the best of our knowledge, there do not exist any comprehensive overviews of such feedback methods for sitting posture. The goal of this work is to provide an overview of the research on hardware to detect sitting posture and of different ways to give the user feedback about it, for which we conducted a literature covering 220 papers. Besides giving a broad overview, we will put more focus on pressure-sensing hardware and visual feedback. Pressure sensors are the most commonly used hardware, while visual feedback is also widely used and incorporates a wide range of techniques such as charts, sketches, physical objects, and more. With the gained knowledge, we then aim to build a smart chair prototype that records sitting posture data, gives visual feedback, and can be used to evaluate such feedback in the future. 15 1 Introduction Through our informal but extensive literature review, we found that there is a large body of work about various types of hardware used for sitting posture recognition, with many approaches combining different sensor types. We concluded that the right hardware depends on cost, privacy concerns, portability, and accuracy. Many publications report high accuracies for automatically classifying postures, for reviews of this aspect refer to Kappattanavar et al. [32] and Tlili et al. [33], as it was out of this work’s scope. Similarly, we found a large body of work exploring feedback about sitting postures, suggesting advantages for all feedback modalities and various types of visual feedback. We argue that multiple modalities and types should be provided and combined as their applicability depends on the environment and the users’ abilities, circumstances, and preferences. Evaluations of visual feedback show generally positive reception by the users and a positive influence on their sitting behavior. Yet, further and more comprehensive user studies are needed to understand the differences between various feedback modalities and types better. Even though our smart chair prototype has some shortcomings, we believe it to be a good starting point for future research into sitting posture recognition and sitting posture feedback. Outline This thesis is structured as follows: Chapter 2 — Related Work: This chapter first gives a brief overview of how much time we spend sitting and the negative outcomes this has on our health. We then outline suggested strategies to mitigate those effects. Chapter 3 - Literature Review about Sitting Posture Recognition and Feedback: Here, we present our literature review that covers hardware used for sitting posture recognition and different modalities used to give feedback about sitting postures. The first part includes a closer look at pressure sensors, while we take a more detailed look at the visual modality in the second part. Chapter 4 — Smart Chair Prototype Design: We give a detailed description of our smart chair prototype in this chapter. This includes the hardware we used and the software we developed to collect, store, and visualize the data. Chapter 5 — Discussion and Limitations: In this chapter, we explain and discuss the limitations of our work and give possible directions for future work. This encompasses the scope of the literature review and the hardware and software of our smart chair prototype. Chapter 6 — Conclusion: In the final chapter, we summarize our work. 16 2 Related Work This chapter presents related publications that serve as a basis for our work. First, we show how often and how long people sit, how this developed in recent years, and what effect the COVID-19 pandemic had on it. We then show that prolonged sitting has various far-reaching negative effects on our health. This is followed up by a look at strategies that have been suggested to mitigate the negative effects of sitting, including the reduction of sitting time, novel office spaces, and computer-aided approaches. 2.1 Sitting and Its Effects on Health We first have to take a look at the terminology around sitting as health and activity research usually explore Sedentary Behavior (SB), a broader term including sitting, reclining, and lying postures with low energy expenditure [34]. We primarily investigate sitting, but we still include studies that investigate SB as findings like the importance of reducing time with low energy expenditure are also applicable to sitting. Whether we are talking about SB or only about sitting, there is a general development towards more sitting-focused lifestyles, but accessing how much time we spend sitting each day is difficult. Gathering such data on a large scale depends on self-reporting of human behavior, which is not as robust and reliable as controlled but size-limited studies. Nevertheless, multiple studies collected and aggregated such data. Bauman et al. [1] analyzed data from 20 countries and found that people sit five hours per day on average. However, the sitting time is influenced by age and education and varies highly between countries. For Germany, Wallmann-Sperlich et al. [2] found that the median sitting time is also five hours per day. Similarly, Mclaughlin et al. [3] report a mean daily sitting time of 4.7 hours for 62 countries. However, Kazi et al. [4] report median sitting times between six and 12.5 hours for workers in education, local government, retail, telecoms, and the service industry in the UK. Long-term studies evaluating data gathered in the EU between 2002 and 2017 revealed an increase in the number of days with more than 4.5 hours spent sitting [5]. The COVID-19 pandemic where we had lockdowns and other restrictions on public life resulted in a further increase of SB and a decrease in physical activity in recent years [8], as measured by, for example, daily step counts [7]. McDowell et al. [6] found the reasons for this increase in SB to be job loss and more time spent working in the home office. Wilms et al. [35] further found, that the increased work in the home office resulted in an increase of SB by 16%. This might be due to the absence of commuting to the workplace and the fact, that our homes are generally smaller than our workplaces, resulting in shorter paths between our workstations and, for example, the fridge or the toilet. Other restrictions due to the pandemic limited outdoor activities and gatherings of large groups of people resulting in more leisure time being spent at home. We assume that this is another reason for the observed increase in time spent sitting during the pandemic. Some of these effects, such as more home office work, might also carry over to post-pandemic times, as we now know 17 2 Related Work that working from home is not only possible for many jobs but also preferred by some employees and companies. That the relationship to home-based work has changed is shown by the study of Rahman Fatmi et al. [36]. They surveyed 17 companies from 12 countries and found that these companies got more flexible regarding working from outside the office. They further report that “the option of working from home is converted from an exclusive perk that managers could choose to give to the few, to a core privilege that all employees feel they are entitled to.” [36] How the relationship between workers and their offices will develop remains to be seen, but as more people are now aware of the home office alternative, at least some will likely prefer it over going to the office in the future. Independent of the pandemic’s influence on our sitting time, the numbers and their upward trend are concerning, as multiple studies have shown the negative effects of extensive sitting. Dunstan et al. [9] describe too much sitting as a “health hazard” that is linked to type-2 diabetes and premature mortality. Further, a meta-analysis by Rezende et al. [11] covering 54 countries found that higher sitting time increases all-cause mortality. Daneshmandi et al. [12] found that prolonged sitting is a possible cause of exhaustion, decreased job satisfaction, hypertension, and musculoskeletal disorder symptoms in office workers’ shoulders, lower back, thighs, and knees. Stamatakis et al. [22] further found that it is associated with all-cause and cardiovascular disease mortality risk for the least physically active adults. This shows, that how much we sit is an important factor for our health and that we have to find ways to either decrease sitting time or lessen its negative effects, topics we will look at in the next section. 2.2 Mitigation Strategies and Countermeasures Finding methods and techniques to reduce the negative effects of sitting is a crucial matter as it would be beneficial to a lot of people worldwide. The most basic approach is to simply reduce the amount of time we spent sitting, possibly replacing it with physical activity. The benefits of this replacement were shown by Ekelund et al. [37], who conducted a meta-analysis about the connection between SB, physical activity, and negative health effects. They found that more physical activity and lower SB are associated with a reduced risk of premature mortality. This is not surprising, though, as physical activity is widely known to be healthy, which is also evident in the literature [38–40], and thus, a widely promoted measure to decrease prolonged sitting is to break it up with physical activity [9, 15–24] but also with standing up [25–27, 41]. This knowledge is also represented in our smart devices, like smartphones and smartwatches, as they all have the possibility to count walked steps and remind us to stand up frequently. However, Stamatakis et al. [22] found that it is unclear which intensity physical activity in such breaks should have and suggested to “move more at any intensity” until there is more evidence. Furthermore, Benatti and Ried-Larsen [17], as well as Black et al. [26], suggest that those factors likely vary from person to person. Additionally, the report by Biddle et al. [21] of an expert think-tank discussing SB shows that there is currently no consensus in the literature on how frequent and long such breaks should be. They suggest, however, that taking a movement break every 30 minutes is likely appropriate despite the lack of clear evidence, while Wong et al. [20] advocate taking breaks every 20 minutes. But frequently breaking up sitting time or replacing it is only possible to an extent, depending on the person, their task, and their environment or workplace, as many of our societies have developed a focus on sitting for a long time. Many workplaces, for example, are designed to have the operator sit 18 2.2 Mitigation Strategies and Countermeasures for extended periods of time and while it might be possible to take frequent breaks from sitting in a regular office, truck drivers could risk meeting their deadlines if they take more than the mandatory breaks, something they would need to do, as they can not stand up and move around while driving. The best ways to support a reduction of our time spent sitting were explored by Lam et al. [42], who conducted an umbrella study on how to most effectively reduce SB. They found that interventions targeting the physical environment reduce SB most effectively, followed by interventions targeting personal behavior, like consultations or apps. One intervention targeting the physical environment they mention is the use of sit-stand desks. They are one example of active workstations which also include less common walking desks or cycling workstations [43]. Sit-stand desks have, however, the additional benefit of enabling the ergonomic adjustment of workstations and thus supporting frequent posture shifts, and with some degree of success, also good postures [44]. Another intervention targeting the physical environment of office spaces is The End of Sitting by Rietveld et al. [45], which combines breaks and regular sitting posture shifts. They present an office space concept without traditional desks but with various geometric shapes that promote unique working positions and frequent switching between them, which also introduces small breaks. Multiple publications have since then studied the effects this approach has on the people that use it. A study by Withagen and Caljouw [46] found that 83% of 18 participants worked in more than one position. The participants also reported that The End of Sitting supported their well-being better than a standard office. Caljouw et al. [47] further found that most positions in the installation resulted in an energy expenditure that was higher than for sitting but not significantly different from standing. They could not find a difference in productivity between sitting, standing, and the positions available at The End of Sitting. Out of 24 participants, 19 experienced at least one position to be more comfortable or equally comfortable as sitting. A further study by Caljouw et al. [48] observed 14 participants over ten weeks while they spent one hour each week at The End of Sitting. The number of position changes during each session was reduced over the weeks but did not reach zero. However, self-reported measurements about task performance, mood, and comfort stayed unchanged. A similar approach by Damen et al. [49] targeted meeting rooms by introducing three furniture pieces for alternative working positions. They conducted a study with 16 participants and reported that the meetings were more effective but that the participants experienced discomfort with some of the introduced objects. Such novel workspaces seem to be a promising approach to increase both well-being and mobility in the offices of the future. It is unlikely, though, that such approaches can be a quick and broad solution to the acute problem due to their high space and cost requirements, while other workplaces with high sitting times, such as vehicles or cashier desks, cannot benefit from this invention. The second highest rated interventions to reduce SB, according to Lam et al. [42], are those that target personal behavior. For example, one can focus on improving the quality of sitting, namely the sitting posture. There are guidelines, such as from the United States National Library of Medicine [28], suggesting keeping the feet on the floor, not crossing the legs, relaxing the shoulders, keeping the elbows close to the body, and supporting the back, thighs, and hips. Further, sitting upright, rather than slumped or forward-leaning, is commonly viewed as a healthier sitting posture [10, 13, 20, 29–31]. However, recent research suggests that the importance lies in the frequent change of sitting postures [10, 21], which is also reflected in the guideline mentioned above [28]. The idea is not new, though, as earlier work by Vergara and Page [50] found that such changes in sitting posture are a good indicator of comfort. 19 2 Related Work Besides spreading this knowledge about sitting postures and healthier sitting behavior to increase awareness of it, continuous feedback about how we sit might be necessary to enable us to act on this information. One approach to give someone feedback about their sitting posture or to suggest breaks is directly from one person to another by manually assessing the sitting posture by someone knowledgeable who can then provide instructions on what should be changed [51, 52]. For example, Dib and Sturmey [51] had an instructor observe the sitting behavior of children playing the flute and give direct feedback in the form of instructions and modeling, which improved their postures significantly. Although very powerful, this approach is cost-intensive and not viable to be deployed widely. While spreading information is essential and in-person training is very effective, these concepts are not countermeasures that can take immediate widespread effect. Further, consistently acting on the knowledge about the negative effects sitting has, especially during day-to-day work, might also be difficult. Another possibility is the detection of one’s sitting posture with sensors and then giving feedback that gives the user information about how they sit and suggests posture shifts or sitting breaks. Such approaches can be applied in a wide range of work environments and could thus reach many of the most affected people, such as office workers or truck drivers that have to sit for long periods each day. Thus, notifying users about poor sitting posture or prolonged sitting through the use of, for example, a smart chair could be an effective way to help many people lead healthier lives. For a large potential user base, costs need to be kept low, while the system should not annoy the user with too many interruptions of their daily work. For example, notifying users based on a short and fixed interval would most likely annoy them and lead to the notifications being ignored or disabled. Thus a more elaborate approach is needed that requires hardware to make informed decisions on when to notify the user based on their sitting behavior. We take a closer look at sitting posture recognition and feedback in the following literature review. 20 3 Literature Review about Sitting Posture Recognition and Feedback This literature review consists of two parts. The first provides an overview of technologies used to detect different sitting postures with a focus on pressure sensors. In the second part, we present different modalities used for posture feedback, with a more in-depth exploration of visual approaches which are the most prevalent. We started our search for related work by manually entering keywords around the discussed topics, such as prevalence of sitting, sitting posture recognition, and sitting posture feedback in various databases like Google Scholar1 and Connected Papers2. We extended this search by looking at referenced publications and citations of the papers we found. We also include works that studied postures that are related to or a part of sitting postures, for example, the angle of the head or back and postures while sitting on the floor or while standing. In total, this review covers 220 publications. 3.1 Sitting Posture Recognition To remind someone to change their sitting posture and take breaks but not annoy them with too many notifications, one has to know how and for how long someone is sitting. Manually scoring sitting postures, as done by Sigurdsson and Austin [53], as well as Sigurdsson et al. [54], or giving in-person training as Dib and Sturmey [51] did, is not feasible on a large scale because of the required time and human resources. To make recognizing sitting postures and giving feedback about them scalable, hardware and automation are required to detect and differentiate between sitting postures. A large body of research exists in this area, of which the following gives a brief overview, with a focus on pressure sensors in the second section. 3.1.1 Wearable, Optical, and Rarely Used Setups One approach is to attach sensors to the user’s body or clothes, such as accelerometers [55– 69], gyroscopes [70–72], or a combination of both [73, 74], see Figure 3.1b for an example. Others used Inertial Measurement Units (IMUs), which combine accelerometers, gyroscopes, and magnetometers [75–87]. Further examples are optical-fiber sensors [88–90], strain sensors [91, 92], flex sensors [93, 94], capacitive proximity sensors [95], and an Electromyography (EMG) setup [96]. Custom sensors that have been studied include a fabric that generates a charge upon being stretched or compressed [97] and a custom angular displacement sensor [98]. Other methods 1https://scholar.google.com/ 2https://www.connectedpapers.com/ 21 https://scholar.google.com/ https://www.connectedpapers.com/ 3 Literature Review about Sitting Posture Recognition and Feedback (a) Feng et al. [99] (b) Hong et al. [70] Figure 3.1: Examples of wearable sensors for sitting posture recognition: RFID tags (a) and a gyroscope (b). requiring something to be attached to the user are Radio-Frequency Identification (RFID) tags [99, 100] (see Figure 3.1a for an example), a neckband emitting ultrasound to measure the distance to a smartphone or computer [101], metallic markers detected by a chair-mounted inductive proximity sensor [102], and electrodes placed on the user’s back, as well as the backrest, to determine the contact area between them [103]. Although these sensors can give high-precision values of, for example, the angle of the user’s back, the fact that they have to be worn may, in our opinion, come with considerable drawbacks regarding their ease of use, acceptance, and deployability. One has to consider weight, size, battery capacity, and how to attach the sensors to the user to make it the least disturbing. Besides wearable sensors, some publications used stationary optical setups. Several articles used a Microsoft Kinect [104–120], while others opted for cameras with various approaches to posture recognition such as face detection [121–124], silhouette extraction [125–127], the use of OpenPose [128, 129], motion capturing [130], and deep learning [131]. While image processing is powerful, it might raise concerns about privacy and confidentiality and has comparably high computational demands. Different optical approaches without those drawbacks include Lidar [132] and depth sensors [133, 134]. Sensors that have rarely been used for sitting posture recognition in the literature include ultrasonic sensors [135], temperature sensors [136], not-worn accelerometers [137, 138], not-worn RFID tags [139], and HTC VIVE Pro trackers [123]. Some publications used combinations of different sensor types, such as accelerometers with a camera [140], a worn tilt sensor with additional ultrasonic sensors [141], and a mix of sensors for temperature and sound [136]. 3.1.2 Pressure Sensors The most commonly used hardware to detect sitting posture is pressure sensors. While we found one instance where a pressure sensor is worn [142], they are generally attached to a chair. Some publications used simple mechanical switches [143–147] and systems that record the shape of the user’s buttocks [148, 149], while others used sensors that change their resistance or charge through deformation due to effects called piezoresistiveness and piezoelectricity, respectively. We will focus 22 3.1 Sitting Posture Recognition (a) Micro load cell by GALOCE3. (b) A sensor matrix stitched onto fabric for pants by Skach et al. [142]. Figure 3.2: Examples of a Hard-PS load cell (a) and a Soft-PS textile sensor (b). on the latter as they have been used widely in the literature to build posture-detecting smart chairs. The terms used to describe them are, unfortunately, inconsistent. We, therefore, introduce two categories to simplify this heterogeneous use of names and to group physically similar sensors. We call the first category Hard-PS and include all sensors with a rigid and rather large body, such as force transducers that can measure positive and negative force, but mainly load cells which are typically used for scales. One example is shown in Figure 3.2a. These use piezo effects by attaching flexible sensors to rigid metal parts, resulting in more force needed to deform them and thus allowing larger weights to be measured. They are, however, uncomfortable when sitting directly on them and are, therefore, usually placed under a plate if they are used for smart chairs. They have been used in smart chairs solely [150–155] or in combination with other sensors, such as inclinometers [156] or sensors for temperature, blood pressure, and pulse [157]. Another type of setup in this category uses sensors that measure air pressure inside bladders on which the user sits [158–162]. The second category we define is Soft-PS and includes all sensors that we identified as thin and flexible, such as flex-, textile-, or fabric pressure sensors and Force Sensitive Resistors (FSRs). See Figure 3.2b for an example. They are very thin in contrast to sensors of the Hard-PS category and can thus be placed anywhere without being noticed by the user. The sensors in this category are the most commonly used sensors for smart chairs, with typical placement options being a chair’s seat or backrest, where they can be placed on top, below, or on the inside of the cushioning. They can also be integrated into a portable pad that does not bind the setup to a specific chair. Some papers we found followed this approach and put such a pad on the backrest of a chair [163–167]. They report promising accuracies for posture classification, while some also observe a limitation due to the necessary contact between the user’s back and the backrest [164, 165]. A more common placement for pressure sensors is the seat of a chair [168–199], for which high accuracies have also been reported. However, most work we found placed them on the seat and the backrest to receive more complete data about the pressure distribution [200–240]. In one case, sensors were placed 3https://www.galoce.com/products/micro-load-cell/GML670A_Half_Bridge_Micro_Load_Cell.html 23 https://www.galoce.com/products/micro-load-cell/GML670A_Half_Bridge_Micro_Load_Cell.html 3 Literature Review about Sitting Posture Recognition and Feedback on the seat, backrest, and, additionally, the chair’s armrests [241]. Cheng et al. [242] followed a different approach by placing pressure sensors below a chair’s legs. We see the most potential in the placement on a chair’s seat, and while very detailed information can be gathered by adding sensors on the backrest, this can only detect postures if the user is in contact with it. Thus, we believe that, depending on the use case, other sensor types could also be used to gather more information about the user’s upper-body posture. As for the previous sensor types combining Soft-PS with other types of sensors has also been studied. Li and Aissaoui [243] combined an FSR mat on the seat with a shape-sensing array below it to reconstruct three-dimensional deformation. Some combined them with optical setups, such as a Microsoft Kinect, such as Ishimatsu and Ueoka [244] who used FSRs under a seat cushion to determine the user’s left-right balance and a Kinect to the side of the user to approximate the back angle. Similarly, Murata and Shibuya [245] used FSRs with a low activation force of 50g together with a Kinect to ensure an ergonomic posture. The Kinect measured if the user’s eyes were at the correct height and the head at an appropriate distance from the monitor. The FSRs were used to detect if the feet touched the ground and the lower and upper back touched the backrest of the chair. One optical approach that did not use a Kinect was followed by Bagalkot et al. [246], who equipped a motorcycle with FSRs and a marker-detecting camera to determine a rider’s weight distribution and upper body posture. Others combined FSRs with sensors to measure acceleration or displacement, such as Zemp et al. [247], who put FSRs in the seat, backrest, and armrests with an IMU in the backrest, while Ma et al. [248] placed both FSRs and an IMU in the seat of a chair. Benocci et al. [249] combined FSRs with an accelerometer, magnetometer, altimeter, and temperature sensor to determine the weight distribution, workload, stress, and the transition between sitting postures. In addition to FSRs on the seat and backrest for sitting posture detection, Hu et al. [250] attached an accelerometer to a chair to measure vibrations that might cause back pain. Another was followed by Ren et al. [251], who added polyvinylidene fluoride film sensors to measure the user’s heart rate variability. Four sensor types were combined by Hong et al. [252]. They attached a gyroscope and an accelerometer to the user’s back to determine if the user leans forward, placed FSRs on the chair’s seat for presence detection, and an infrared sensor on the computer monitor to measure the distance to the user’s head. Multiple publications used sensors to measure the distance between the user’s back and the backrest to differentiate upper body postures such as leaning forward or sitting upright. For example, multiple papers attached an infrared sensor to the backrest in addition to FSRs on the seat [253, 254] and both the seat and backrest [255]. Others used ultrasonic sensors on the backrest with FSRs on a chair’s seat to differentiate different sitting postures [256–258]. This approach is supported by the analysis of Ordean et al. [259] and has also been applied to a wheelchair by Rosero-Montalvo et al. [260]. Further, Li et al. [261] used multiple ultrasonic sensors at different heights on the backrest while also deploying FSRs on a chair’s seat. 3.1.3 Evaluation While many of the above-mentioned publications report considerably high accuracy, comparing them is outside the scope of this work and, so we believe, will prove to be a difficult task nonetheless, as many fundamental aspects of these systems are very heterogeneous. This is evident in the various 24 3.2 Sitting Posture Feedback types and amounts of defined postures, as they range from the differentiation between good and bad (e.g., [86]) to the definition of 30 individual postures [116]. A great variation can also be seen in the means to classify sitting postures. They range from comparing sensor values to thresholds (e.g., [257]) to various machine learning approaches (e.g., [192, 195]). For an extensive review of sitting posture monitoring systems and different classification approaches, we would like to refer the interested reader to the reviews by Tlili et al. [33] and Kappattanavar et al. [32]. 3.1.4 Conclusion Our overview demonstrates the large variety of approaches and sensors that have been studied in the context of sitting posture recognition. This not only reflects the interest of governments and scientists in the topic and its importance to public health but also that there is room for various concepts to co-exist. We conclude that depending on requirements on cost, privacy, portability, and accuracy, all the different hardware approaches are more or less applicable. Wearable sensors offer great portability and are thus not bound to a specific setup or chair. But as they have to be attached to the user’s body or clothes, one has to consider factors such as the sensors’ size, weight, and the method of attachment, to not disturb the user. Optical setups can utilize the possibilities of image processing, but these come with comparably high computational costs, while the use of cameras might raise privacy concerns. For other approaches like the use of temperature sensors, HTC VIVE Pro trackers, or combinations like tilt and ultrasonic sensors, more research is needed, in our opinion, to gain knowledge about their usability and performance. Pressure sensors are the most used technology in the literature, with flat and unobtrusive types, which we called Soft-PS, being the most prevalent. Other types we referred to as Hard-PS include load cells and sensors that measure the pressure inside air bladders. Sensors of the Soft-PS category, like FSRs can be deployed on a chair’s seat, backrest, and armrests or in a separate pad that can be used with different chairs. We see the greatest potential in combining pressure sensors in a chair’s seat with other sensor types, such as ultrasonic distance sensors, to capture the user’s upper-body posture. 3.2 Sitting Posture Feedback As the previous section shows, a large body of research has been conducted on various technologies and techniques to detect and classify sitting postures. Giving users feedback about their posture has also been studied extensively. Researchers explored hardware that actively adjusts itself to directly or indirectly correct the user’s posture, electrical stimulation, as well as vibrotactile, aural, and visual modalities. While all of these approaches have their advantages, visual feedback is the most prevalent and varied approach in the publications we found. However, non-visual modalities could, according to the multiple resource theory by Wickens [262], actually be beneficial for scenarios where the main task of the user is highly visual, such as most office work. Furthermore, people with decreased mobility might benefit more from systems that can actively change their sitting posture. This review will cover all of these approaches to map the research on sitting posture feedback. We will, however, focus more on visual feedback, especially regarding user studies, because of the already mentioned prevalence and variety of this modality and the authors’ expertise with visualizations. 25 3 Literature Review about Sitting Posture Recognition and Feedback (a) Shen et al. [238] (b) Shin et al. [123] (c) Fujita et al. [191] Figure 3.3: Examples of actively correcting the user’s sitting posture: inflatable bladders (a), self-adjusting computer display (b), and self-adjusting chair and desk (c). 3.2.1 Active Correction, Aural, and Vibrotactile Feedback Besides giving visual feedback about sitting posture, which we will discuss later, other modalities than visuals have also been studied, such as aural (e.g., [85, 130]), vibrotactile (e.g., [74, 194]), and feedback physically influencing the users’ sitting posture (e.g., [123, 238]). See Tables A.1 and A.2 in Appendix A for a complete list of publications that explored such approaches. One example of feedback that actively corrects the user’s posture is that of Kiran et al. [85], who used Electrical muscle stimulation (EMS) to cause involuntary muscle contraction. Another approach by Ishimatsu and Ueoka [107] consists of a system that gives physical feedback by pushing wooden beads attached to sticks up the user’s back. Both these publications include studies comparing their feedback methods to visual feedback, which are explained in more detail in Section 3.2.2.5. A further approach for active sitting posture correction is the use of bladders that can be inflated or deflated to improve the user’s posture [147, 158, 159, 200, 238, 257], of which one example is shown in Figure 3.3a. Other researchers built systems that adjust the user’s workstation to influence their posture directly or indirectly. One approach is to move the computer monitor [123, 263] (see Figure 3.3b for one example) or the content in a Virtual Reality (VR) environment [264] to get the user to adjust their posture. Fujita et al. [191] followed a different approach and built a chair that can change the angle of its seat, shown in Figure 3.3c. Wu et al. [115] used a Microsoft Kinect to measure the user’s dimensions and calculate the optimal chair and desk height and optimal positions for the keyboard, chair, and monitor. Using additional hardware that can actively change how someone is sitting is, in our opinion, the most elaborate approach to giving feedback about sitting posture. We assume that such methods have disadvantages due to cost and size when compared to other methods, while we also see a great advantage of them because they can improve the user’s posture without their attention. This might be crucial, for example, if the user needs to keep focused on their current task, such as driving a vehicle. Other non-visual feedback modalities are the use of sound and the use of vibration, namely aural and vibrotactile, respectively. We assume the possibility of other people hearing the generated sound by such systems to be a potential drawback. This is possible if aural feedback is given through speakers or if the actuators generating vibrotactile feedback are mounted in a way that they also produce audible sound, for example, to the wooden board of a chair. We assume that these sounds might disturb other people, such as coworkers, or that the user could be uncomfortable if other people know of their need for feedback about sitting. Regarding the publications we found, most aural feedback was provided through simple sounds (e.g., [73, 86]), while others gave verbal instructions 26 3.2 Sitting Posture Feedback or warnings in person [51, 130, 265] or via recordings [65, 85, 184, 207]. Vibrotactile feedback was given through a single actuator (e.g., [57, 92]) or with multiple actuators to be able to focus the area where a deviation from a good posture was detected (e.g., [164, 165, 209, 255]). 3.2.2 Visual Feedback Visual sitting posture feedback is the most prevalent approach in the publications we found with a wide range of different types, such as ambient lights, text, sketches, charts, and more. Various studies were able to show that such feedback can improve users’ sitting posture, while some compared visual feedback to other modalities. This section gives an overview of previously suggested visual feedback and their evaluations. We only include publications that provide enough details about their visual feedback to allow comparison with other approaches. We identified two main categories in previous work on visual feedback for sitting posture: time and type. The time of delivery can either be in real-time, giving feedback on the current situation or as a summary of how the user sat during a certain period. The different types we found were text messages, sketch-like depictions, charts, images or videos, physical objects such as ambient lights, and other types we summarized as such. We first take a look at the feedback given as a summary, then at the use of physical objects, followed by a more in-depth look at non-physical real-time feedback, as this is the most explored and versatile. See Tables 3.1 and 3.2 for an overview and Table 3.3 for statistics about the publications we found. 3.2.2.1 Summaries One approach is to give the user feedback about their sitting posture in the form of a summary after a certain amount of time. The most used method in the literature for this is the use of dashboards with various types of charts. See Table 3.3 for more details about how often different types of visual feedback were used for summaries. Some used bar charts (e.g., [78, 135]), line charts (e.g., [60, 77, 147, 157]), area charts [147], and pie charts (e.g., [95, 256]), while others used dial charts to show the time spent in different postures [233] and the health-risk level of the user [106]. Some examples of these chart types can be found in Figure 3.4. Furthermore, heatmaps were used to visualize the pressure distribution [147, 215], with two cases using LEDs on a sketched chair that were attached to the side of the chair [211, 213], as shown in Figure 3.5a. Bagalkot et al. [246] created a rounded star plot, which is shown in Figure 3.4c, where each axis represents a characteristic of the sitting posture, such as leaning left. They describe this as an “amoeba-like blob” with the goal of easy readability at a glance while riding a motorcycle. Other types than charts were also used, namely text messages and sketch-like depictions. An example of the prior is by El-Sayed et al. [156], who sent daily textual reports as emails to the user’s doctor for review. Examples of the use of sketches are the stick figures by Ribeiro et al. [160] which depict different sitting positions and how much sitting time the user spent sitting in them, while Yu et al. [145] showed a sketch of a person sitting at a desk with circles at the sensor positions. Those circles were colored green if the respective sensor value was scored as being at risk during a certain time frame. Wang et al. [225] followed another approach and combined sketches with charts by augmenting pie and bar charts with depictions of different postures. 27 3 Literature Review about Sitting Posture Recognition and Feedback (a) Cho et al. [256] (b) Anwary et al. [233] (c) Bagalkot et al. [246] Figure 3.4: Examples of visual feedback summarizing information about the user’s sitting posture: using a bar chart for time spent in different postures and pie charts for posture balance (a), showing time spent in different postures in a dial chart (b), and visualizing multiple sitting posture parameters as a rounded star plot (c). 3.2.2.2 Physical Objects Further, visual feedback can also be provided through physical objects other than displays, ranging from simple LEDs to complex objects that can deform according to the user’s posture. One such technique is data physicalization, defined by Jansen et al. [266] as “a physical artifact whose geometry or material properties encode data.” An early approach by Daian et al. [207] introduced a physical agent on the desk, which turned its back to the user if an inappropriate posture was detected and moved from side to side to suggest a break. Haller et al. [151] created a physical flower that can be deformed to imitate the user’s posture or shake itself to motivate them to do a training session. Later, Hong et al. [252] built upon this approach with a flower, shown in Figure 3.5c, that can imitate the angle of the user’s back while changing the color of its stem from green to yellow as an analogy of poor health. Blue LEDs inside the stem light up to represent the sitting time. This way, the flower slowly becomes the same color as the flower’s pot, which is intended to look like the flower becoming an inanimate object, a metaphor for the stationary user. LEDs start blinking to warn the user of a too-short distance to the monitor, with additional sounds being played if the distance gets even closer. Finally, purple LEDs shine to suggest a short stretching session. Regarding such approaches’ hardware complexity and visibility to other people, they are more closely connected to the modalities we presented as non-visual. Nevertheless, the examples used for sitting posture feedback we found work visually and are thus included in this section. Other approaches using physical objects we found use ambient light. Most of them attach LEDs somewhere on the desk, which light up or blink to give feedback about improper posture [106, 135, 141, 257]. Some put LEDs on a sketched chair that was attached to the side of the chair and used them to display the current pressure values, as shown in Figure 3.5a [211, 213, 220]. Lee et al. 28 3.2 Sitting Posture Feedback (a) van der Doelen et al. [211] (b) Soltani Nejad [267] (c) Hong et al. [252] Figure 3.5: Examples of visual feedback using physical objects: attaching LEDs to a chair (a) or a shirt (b) and creating a physical flower that mirrors the user’s posture (c). [254] encased lights in an ambient display shaped like a cloud and moon and placed them next to the computer display. The two elements glowed dimly if the user sat in a low-risk posture and flashed red if in a high-risk posture. A different approach was developed by Ren et al. [251], who put lights on the underside of a monitor stand. Based on the sitting time and the user’s heart rate, the brightness and saturation of the lights were increased, respectively. They could also pulse to guide a deep breathing exercise and suggest movement by lighting up partially. The physical flower of Hong et al. [252] also uses ambient lights, as described above. Others integrated LEDs into the clothes of the user. Özgül and Patlar Akbulut [74] attached an LED to a vest, while Nishida and Tsukada [77] sewed LEDs into the sleeves of a sweater and Soltani Nejad [267] into the sleeves and the front of a shirt, as shown in Figure 3.5b. Another approach was followed by Wölfel [112], who projected feedback onto a wall in front of the user. They used an anthropomorphic flower that imitates the user’s posture. Even though they clearly work visually, ambient lights and projections are, in our opinion, more comparable to aural and vibrotactile feedback in terms of privacy, as their light could easily be seen by other people around the user. 3.2.2.3 Non-physical and Real-Time We now turn our focus to the most prevalent combination of real-time feedback that does not use physical objects besides displays. The most used type of real-time visual feedback was sketch-like depictions. See Table 3.3 for more details about how often different types of visual feedback were used for real-time feedback. Some approaches made use of images and videos to provide visual feedback. For example, Taieb-Maimon et al. [52] showed the user a picture of their current sitting posture next to a previously taken reference picture after a fixed time. Sigurdsson and Austin [53] and Sigurdsson et al. [54] showed the user live video footage of themselves through which they had to score their posture. Another approach was followed by Taylor et al. [105], who used a large screen to act as a mirror, as shown in Figure 3.6a. The live video was then augmented by highlighting the parts of the user’s body that deviated from good posture or, as a more general feedback, by displaying fog. Text messages have also been used to give prompts, suggesting the user change their posture, take a 29 3 Literature Review about Sitting Posture Recognition and Feedback (a) Taylor et al. [105] (b) Luna-Perejón et al. [193] (c) Kiran et al. [85] Figure 3.6: Examples of real-time visual feedback: a smart mirror with body parts highlighted that deviate from good posture (a), a sketched chair with a heatmap of the pressure (b), and a desktop notification (c). break, or exercise (e.g., [85, 135, 199, 217]). One example of this is shown in Figure 3.6c. More specific suggestions on how to improve the current posture were also given (e.g., [78, 146]), as well as encouraging messages for sitting with a good posture [104]. Multiple publications explored the use of charts to give real-time feedback, including straightfor- ward approaches such as bars being colored green or red depending on muscle activation [96, 130] or lines that are oriented according to the current angle of the user’s lower and upper back [76]. Others used progress bars showing sitting and break time [124], a line chart showing how much the shoulders are bent [86], and dial charts displaying the asymmetries of the current posture [231]. Jaimes [125] displayed a red and green bar over which a black bar moved, representing the user’s left-right balance. There is also an instance of a scatterplot being used, where they used scaled circles representing the pressure values of the sensors [225]. Comparatively often used were heatmaps of the current pressure values [174, 256], with Wang and Yu [215] creating a three-dimensional heatmap in the form of the chair. Many publications used sketch-like depictions to visualize their sitting posture feedback. Kim et al. [268] displayed a turtle with a bent neck, referring to the "turtle-neck syndrome", which is how sitting with a forward bent neck is referred to in South Korea. Others used sketches of chairs with additional information, such as a color-changing background [229], pressure distribution percentages [233], or at-risk positions [145]. Demmans et al. [269] describe a face icon that is, depending on the posture, either green and smiling or red and crying, while Lee et al. [254] showed either a human sitting upright or hunching. Sketches of different postures have also been used, such as by Breen et al. [56], who showed the user their current posture and a red circle if it was deemed bad. Zheng and Morrell [255] used sketches to show cues for how to improve the current posture and sketches of a human back and legs with colored circles where posture errors were detected. Further, Elsayed et al. [270] explored different three-dimensional models of the human body with varying accuracy, namely a skeleton, a silhouette, and a 3D avatar. A virtual skeleton has also been used by Baptista et al. [113], who showed the user their current posture, as well as a suggested posture with arrows indicating the necessary movements to reach it. Using a digital flower to imitate 30 3.2 Sitting Posture Feedback (a) Khurana et al. [271] (b) Hong et al. [70] Figure 3.7: Examples of real-time visual feedback: an anthropomorphized giraffe with information and suggestions about the user’s sitting posture (a) and an anthropomorphized flower enhanced through gamification (b). a user’s posture was explored by Haller et al. [151] and Hong et al. [70]. Haller et al. [151] also created a physical variant, as described in Section 3.2.2.2, while the anthropomorphized flower by Hong et al. [70] is shown in Figure 3.7b and described in Section 3.2.2.4. Charts with sketch-like depictions were also used in combination, such as a dial chart for back and head angle, which is, upon reaching a threshold, colored red, and a bell appears to alert the user [75]. Min et al. [176] combined status bars with a cartoon dog that has different states depending on the users sitting posture, which is described in more detail in the following section. Flutur et al. [95] used a sketched human sitting on a chair with circles corresponding to the sensors. These circles’ colors change based on the sensors’ states, which were inactive, correct, moderate, and incorrect. Multiple researchers combined a sketch of their chair with a heatmap displaying pressure distribution [177, 193, 232], of which one example can be seen in Figure 3.6b. Charts have also been combined with images and videos, and text messages, such as Ishimatsu and Ueoka [107], Jaimes [125], and Ishimatsu and Ueoka [244], who took the approach of representing the user’s posture by angled lines that are displayed over live webcam footage. Or like Davis and Kotowski [272], who showed a progress bar of the user’s idle time and messages reminding them to adjust their sit-stand desk or to stand up and move around. Some publications show a combination of sketch-like depictions with text messages. One example is the approach by Özgül and Patlar Akbulut [74], who showed cartoons about good and bad postures together with descriptions. Another one by Khurana et al. [271] showed an anthropomorphized giraffe whose neck angle and facial expression resemble the user’s posture and posture quality. They, additionally, displayed general information about sitting posture and suggestions on how the user can improve theirs. While multiple publications showed sketches of a person on a chair together with some information [90, 93, 245], Murata and Shibuya [245] added red circles around zones for which a bad posture was detected and provided additional information on how to correct these. Nizam et al. [93] showed arrows suggesting posture changes together with a text explanation. The sketched human of Tavares et al. [90] adopted different postures while a text told the user that their stance was incorrect or suggested taking a break. 31 3 Literature Review about Sitting Posture Recognition and Feedback Some publications combined more than two of the visual feedback types we defined, such as Ochoa et al. [65], who used an image of a human spine and added colored text labels for parts of the spine if the sensor at the corresponding position detected a bad posture. Further, Shen et al. [238] created a heatmap of the pressure distribution, a bar chart of the sensors’ pressure values, a sketch of a human representing the user’s current posture, and a text message that encourages the user to do exercises or relax. Finally, Speir [217] drew colored circles at the sensor positions on an image of their chair, using red for sensors that showed a deviation from the reference posture. An additional text suggested that the user should change their posture. 3.2.2.4 Other Methods Other methods of non-physical real-time feedback were studied by Duffy and Smeaton [104], who dimmed the monitor’s brightness if the users sat in a bad posture, while oOthers flashed the computer- [107] or smartphone- [60, 63] display to alert the user of a bad sitting posture. Another approach by Shin et al. [61] introduces “Relational Norm Intervention”, which uses negative reinforcement and the desire not to disturb others. They, therefore, introduce a second person they call a “helper”. If the user sits in a bad posture and does not change their posture after receiving a vibrotactile notification, the helper’s phone gets blocked. The helper can then send a push notification to the user, optionally with a text message. A different method that has been used in some cases to enhance the effects of sitting posture feedback for the user is gamification. Blohm and Leimeister [273] described gamification as “enriching products, services, and information systems with game-design elements in order to positively influence motivation, productivity, and behavior of users”, which is being used in many areas such as education [274] and retail [275]. Khurana et al. [271] used gamification in the form of badges that can be earned, such as “exercise your neck for 3 minutes”, visible in Figure 3.7a. Murata and Shibuya [245] used a posture score, i.e., the proportion of time spent in a good sitting posture in the last hour, and a ranking comparing this score with that of others. Min et al. [176] show the user a cartoon dog that is influenced by the user’s sitting behavior and has to be kept healthy. They use status bars that visualize parameters of the user’s sitting posture, such as a bar for the dog’s hunger, which decreases if the user leans to the right side too much. If these bars decrease to a critical level, the dog either blinks, rotates its head, or pants, depending on the decreased bar. While the user is performing countermeasures, the dog is animated accordingly. In the above example, leaning left after leaning to the right side and decreasing the hunger bar starts a feeding animation of the dog. Hong et al. [70] used gamification in the form of points that can be used to customize an anthropomorphized flower, badges that can be earned, and levels. Some of these features can be seen in Figure 3.7b. The system lets the user take care of the flower through proper sitting. Additionally, suggestive missions unrelated to sitting, such as cleaning the room or drinking water, were integrated. Finally, users can put fully grown flowers into a garden where they show statistics, and the user can start a new flower. 3.2.2.5 User Studies In this section, we give an overview of publications evaluating visual sitting posture feedback. We want to outline findings in the area, but also the types and sample sizes of conducted studies. Our goal is to provide a guide to possible future work by identifying aspects that need to be investigated 32 3.2 Sitting Posture Feedback Publication Year Time Type RT SUM IV TM CH SL PO Other Jaimes [125] 2005 × × × Demmans et al. [269] 2007 × × Daian et al. [207] 2007 × × Sigurdsson and Austin [53] 2008 × × Breen et al. [56] 2009 × × Mu et al. [121] 2010 × × van der Doelen et al. [211] 2011 × × ⊗ El-Sayed et al. [156] 2011 × × ⊗ Goossens et al. [213] 2012 × × ⊗ Haller et al. [151] 2011 × × × Sigurdsson et al. [54] 2011 × × Park and Yoo [96] 2012 × × Taieb-Maimon et al. [52] 2012 × × Duffy and Smeaton [104] 2013 × × × Moon and Oah [146] 2013 × × Taylor et al. [105] 2013 × × Wang and Yu [215] 2013 × × × ⊗ Xu et al. [174] 2013 × × Yu et al. [145] 2013 × × ⊗ Zheng and Morrell [255] 2013 × × Alattas and Elleithy [135] 2014 × × × # × Davis and Kotowski [272] 2014 × × × Ishimatsu and Ueoka [244] 2014 × × × Khurana et al. [271] 2014 × × × # × # Paliyawan et al. [106] 2014 × × × # × van Almkerk et al. [147] 2015 × # Hong et al. [70] 2015 × × × Hong et al. [252] 2015 × × Ishimatsu and Ueoka [107] 2015 × × × × Min et al. [176] 2015 × × ⊗ × ⊗ Ribeiro et al. [160] 2015 × # Speir [217] 2015 × × × × Wang et al. [75] 2015 × × × Gaffney et al. [130] 2016 × × Kim et al. [268] 2016 × × Table 3.1: Overview of publications containing visual feedback for sitting postures. The time of delivery is separated into real-time (RT) and summary (SUM). The visual types are images or videos (IV), text messages (TM), charts (CH), sketch-like depictions (SL), physical objects (PO), and other types, such as gamification. For the types, × denotes their use for real-time, # for summary, and ⊗ for both times of delivery. See Table 3.3 for statistics about these publications. 33 3 Literature Review about Sitting Posture Recognition and Feedback Publication Year Time Type RT SUM IV TM CH SL PO Other Liao [60] 2016 × × # × Murata and Shibuya [245] 2016 × × × # × # Park et al. [177] 2016 × × × Shin et al. [61] 2016 × × Baptista et al. [113] 2017 × × Liao [63] 2017 × × # × Nishida and Tsukada [77] 2017 × × # × Petropoulos et al. [76] 2017 × × ⊗ Roossien et al. [220] 2017 × × ⊗ Wölfel [112] 2017 × × Ochoa et al. [65] 2018 × × × Soltani Nejad [267] 2018 × × Wang et al. [225] 2018 × × ⊗ Anwary et al. [231] 2019 × × Bagalkot et al. [246] 2019 × # Bootsman et al. [78] 2019 × × × # Cho et al. [256] 2019 × × ⊗ Flutur et al. [95] 2019 × × ⊗ × Prueksanusak et al. [229] 2019 × × # × Ren et al. [251] 2019 × × Wang and Reiterer [124] 2019 × × × Anwary et al. [233] 2020 × × # × Lee et al. [254] 2020 × × × Matuska et al. [232] 2020 × × × Nizam et al. [93] 2020 × × × Elsayed et al. [270] 2021 × × Kiran et al. [85] 2021 × × Kumar and Sridhar [157] 2021 × × Lee et al. [257] 2021 × × Luna-Perejón et al. [193] 2021 × × × Ramalingam et al. [141] 2021 × × Shen et al. [238] 2021 × × × × Tavares et al. [90] 2022 × × × Tlili et al. [86] 2022 × × × Lamberti et al. [199] 2022 × × Özgül and Patlar Akbulut [74] 2022 × × × × Table 3.2: (Cont.) Overview of publications containing visual feedback for sitting postures. The time of delivery is separated into real-time (RT) and summary (SUM). The visual types are images or videos (IV), text messages (TM), charts (CH), sketch-like depictions (SL), physical objects (PO), and other types, such as gamification. For the types, × denotes their use for real-time, # for summary, and ⊗ for both times of delivery. See Table 3.3 for statistics about these publications. 34 3.2 Sitting Posture Feedback Type Number of times used for Real-Time Summary Images or Videos 9 0 Text Messages 20 1 Charts 22 18 Sketch-like Depictions 26 2 Physical Objects 16 3 Other 7 3 Table 3.3: The number of times different visual types were for sitting posture feedback by the publications listed in Tables 3.1 and 3.2. in more detail. We, therefore, order this section according to the number of participants and the duration of the studies. We also excluded four publications due to a lack of quality in the description of the study or the analysis of the results. Less than 24 participants and a duration of less than one day Daian et al. [207] conducted a user study over five to six hours with six participants about their combination of a physical agent and recorded aural recommendations. Although there is no statistical analysis, the authors report some findings based on video recordings and semi-structured interviews. The participants had an overall positive perception of the chair and responded quickly to the feedback by changing their sitting posture. Sigurdsson and Austin [53] gave eight participants information about sitting posture and live video footage of themselves, based on which they had to rate their own sitting posture during the day. While they found the intervention, which happened every 55 seconds, to improve their posture, they also noted a mean decrease in productivity by 11%. Ishimatsu and Ueoka [244] carried out a study with eight participants to test their sitting posture feedback. When the participants’ posture deviated from a previously recorded reference posture, a pop-up window appeared. In this window, a live video of the participant is shown with two lines overlaid, one representing their current and one the reference posture. They compared their feedback with a baseline condition where no feedback was given, both lasting 30 minutes. The participants showed a significant decrease in instances of bad posture for the feedback condition. Ishimatsu and Ueoka [107] invited 12 participants to a user study comparing visual feedback in the form of flashing the screen to pushing a wooden bead on a stick up the participants’ backs. During both conditions, which lasted 30 minutes each, a window in the corner of the screen showed a live video of the participant with two lines that represented the current and the reference posture. They report that the flashing of the screen was not noticed every time and that some participants reported that they adjusted their posture involuntarily based on the physical feedback instead of the lines displayed over their video. Haller et al. [151] conducted a study over 1.5 hours with 12 participants to compare vibrotactile feedback with a physical flower imitating the user’s posture and a graphical representation of the flower on the computer display. While the users performed one of three tasks with different difficulties, the feedback prompted them to start a training session. This was repeated every 30 seconds if they did not start it within 15 seconds. They found that the graphical feedback resulted in a longer time needed to start the training and to resume the main task. The participants 35 3 Literature Review about Sitting Posture Recognition and Feedback reported that the physical flower interrupted their workflow the least. Vibrotactile feedback got rated as the most disturbing for continuous feedback and the physical flower as the least disturbing for training reminders. Park and Yoo [96] conducted a study with 14 participants to compare the effect of visualizing the activation of different muscles on head and back angles as an indication of good posture. Values of four different muscles were visualized, resulting in four conditions of which each lasted 15 minutes. They visualized values below or above a threshold with a bar being colored green or red, respectively. The baseline without feedback resulted in significantly worse angles of the head compared to all conditions with visualizations and for two out of three in the case of the back. Wölfel [112] let 16 participants experience their system that projected an orchid on the wall next to the computer display. The orchid imitated the user’s posture by moving its leaves and bending or leaning the stem. The flower’s “face” would turn “sad” if the user was sitting in a bad posture. The participants used the system for three hours while researching something on Wikipedia, after which they filled out a questionnaire. The responses indicated that the participants understood and liked the visualization, while the responses were mixed about their desire to use the system regularly. Bootsman et al. [78] tested their system of a sensor-equipped shirt targeted at nurses. It gives either aural and vibrotactile feedback or additional smartphone notifications, which ask the user to add information about their current activity with the goal of increasing awareness over time. In an initial study, 15 nurses wore the shirt for one hour per feedback condition and found it comfortable. However, no differences were found between feedback types in terms of perceived credibility and motivation. They conducted a second study over 3.5 hours with 13 different female nurses to gauge the system’s effectiveness in improving posture. They found improvements for both feedback conditions, but could, however, not discern a superior method. Less than 24 participants and a duration of more than one day Dib and Sturmey [51] studied three children getting verbal instructions, modeling the correct posture, rehearsal, and feedback while playing the flute. Video recordings of the training sessions were scored manually and showed a great improvement in sitting posture, which did not decrease even after a one- to two-month follow-up. Moon and Oah [146] had three participants from their university administrative staff sit on a smart chair for four days and measure the time they spent in a reference posture. They first sat on the chair without receiving feedback, after which they got briefed on safe sitting postures. In the second session, they were shown text messages explaining which part of their body was not in line with the reference posture. The third session gave text messages with information about the risks and benefits of bad and good postures, respectively. They were delivered independent of the posture with the same frequency as those of the second session. For the fourth and final session, they received the same feedback as in session two. The authors report that the participants spent the least amount of time in the safe sitting postures during baseline and the most during the body-part-specific feedback. Duffy and Smeaton [104] conducted a study with four participants over four days and compared three different feedback methods with a baseline where no feedback was given. They either dimmed the monitor’s brightness, showed a pop-up showing the time spent in different postures, or a pop-up with encouraging messages. Both pup-up types were shown to the participants once per hour while the screen was dimmed after a certain amount of time spent in a bad posture. They report that dimming the screen resulted in the best results for time spent in good posture and for continuous time spent in good sitting posture. Over the time of the study, all participants showed a decrease in 36 3.2 Sitting Posture Feedback continuous time spent in bad postures. Yu et al. [145] conducted a study over 15 days with four office workers to compare real-time with summary feedback. The feedback had the form of a pop-up window with a sketch of a human sitting at a desk with circles at the sensor positions, which were colored green if the sensors indicated the violation of a good posture. The summary feedback colored all circles with violations during a session and showed their amount. One condition received only the summary, while the other participants additionally received real-time feedback. They report an increase in the time spent in good postures for both conditions, with better results for the combination of feedback methods. They could, however, not isolate the effect of the real-time feedback due to their study design. Murata and Shibuya [245] investigated the effect of their feedback on the users’ awareness of bad posture and their motivation to maintain a good posture through a user study with six participants that was conducted over four weeks. The feedback showed a sketched person on a chair with information about how the user’s posture deviated from a good one. Two further conditions included a leaderboard of multiple users’ posture scores, once visualized through progress bars only and once with sketched humans for each user. Their results indicate that the base feedback and the leaderboard with sketches decreased the time spent in bad posture significantly. A questionnaire revealed that all feedback types made the participants more aware of poor posture and the parts of the body that deviated from good posture, while they reported no significant difference regarding interruption. Shin et al. [61] studied their “Relational Norm Intervention” approach with six teams of two that used the system over two weeks. Each duo had a “helper” whose phones got blocked and could then send a push notification to the other user if they sat in a bad posture and did not react to a vibrotactile notification. The user only received the initial notification for the control condition without incorporating the still-present helper. The researchers found their approach to be more effective in correcting posture compared to the control intervention. They identified the main reason to be the users being uncomfortable with disturbing the helpers, who were not bothered by it. Wang and Reiterer [124] conducted a three-week-long field study with eight participants to compare text pop-ups suggesting a break with an always-on progress bar showing the remaining time of the current state, either work or break. The progress bar was more popular with the participants, while the pop-ups were more effective in reducing sitting time. At least 24 participants and a duration of less than one day Lee et al. [254] compared a desktop notification showing a sketched human that either sits upright or hunched with an ambient display shaped like a cloud and moon that flashes red if a high-risk posture is detected. A study with 24 participants was conducted comparing the two types of feedback to a baseline without notifications. Each participant had to solve arithmetic problems for 20 minutes for each condition. Their results show a significant decrease in high-risk postures for both types of feedback compared to the baseline while also increasing the number of typed answers. The participants found the ambient display to be more visible and understandable than the desktop notification. The users had a more positive attitude toward the ambient display and expected a better performance, while no difference in expected effort and social influence could be found. Speir [217] compared two visual feedback methods with different amounts of information through a user study with 34 participants that lasted for 75 minutes. The feedback either only prompted the participant to adjust their posture through text or additionally provided an image of their chair with red circles indicating deviation from good posture. The authors report participants preferred more detailed feedback, while they could not find a significant difference between the conditions. 37 3 Literature Review about Sitting Posture Recognition and Feedback Kiran et al. [85] compared text messages and aural feedback to automatic and involuntary correction of slumped sitting posture through Electrical Muscle Stimulation (EMS) applied through electrodes attached to the user’s upper back. A user study with 36 participants was conducted over 75 minutes. The authors report that the time to correct the sitting posture was significantly lower for EMS while also being more accurate, providing similar levels of comfortability and disruption. Zheng and Morrell [255] compared visual and vibrotactile feedback in two studies. The visual feedback showed cues to improve the posture and areas of the body where posture errors were detected. The first study measured the adoption compliance of a reference sitting posture of 25 participants during two 45-minute sessions for vibrotactile feedback and one 45-minute session for visual feedback. The second study focused on cognitive load with 41 participants, measured during four five-minute long sessions. Both studies showed no significant differences between the two types of feedback. At least 24 participants and a duration of more than one day Davis and Kotowski [272] studied 37 call center employees’ discomfort, postural variability, and productivity between four conditions over four weeks. The workers either had a conventional workstation or a sit-stand desk, either with or without reminder software. The software showed a progress bar of the participants’ idle time and a message telling them to switch between sitting and standing or that they should get up and move around. The researchers found that both interventions and their combination increased movement and decreased comfort at the end of the day while not impacting productivity. Taieb-Maimon et al. [52] studied 50 university and university hospital employees. They compared sitting posture development over six weeks between a baseline group with one that was provided training and workstation adjustments and one that also received visual feedback. Sitting postures were semi-automatically scored based on images. At regular intervals, the participants in the feedback group were shown an image of themselves and one where they sat in a reference posture. Their analysis shows improvements for both conditions in the beginning, with only the visual feedback group maintaining this improvement until the end of the study. The authors, therefore, suggest the need for repeated reminders. 3.2.3 Conclusion We found that researchers explored various modalities for sitting posture feedback, including sounds, vibration, visual feedback through displays, physical agents, or blinking lights, and hardware or furniture that moves to directly or indirectly influence the user’s posture. We believe that all these approaches have possible application areas and situations, while their applicability depends on the environment and the users’ abilities, circumstances, and preferences. Physical agents could be appealing to a younger user group, while sound might be beneficial for people who can wear headphones but have limited desk and screen space. Visual feedback provided through displays can be shown on various kinds of devices, from computers to smartphones, and can deliver a lot of information. On the other hand, vibrotactile and aural feedback might be beneficial if users tend to miss visual feedback due to other visual stimuli overburdening them, as suggested by the multiple resource theory by Wickens [262]. The presence of other people that might get disturbed by sounds or flashing lights can limit the possibilities, as can hearing impairment or privacy concerns of the user. While more expensive solutions, such as self-adjusting computer displays, might be financially unfeasible in many cases, people with decreased mobility could benefit from active systems such as self-inflating bladders (e.g. [238]). To develop a successful system, one should factor in all this 38 3.3 Summary and Findings information to provide a system that fits the targeted user group. We think that giving users the ability to choose and combine multiple modalities according to their preferences is necessary to create a satisfying and motivating experience. Focusing more on visual feedback, we identified two times of delivery and five different types. Most works focused on giving the users feedback about their sitting postures in real-time, while some provided summaries after different periods. The types we identified are text messages, sketch-like depictions, charts, images or videos, physical objects, and other methods like gamification, as shown in Tables 3.1 and 3.2. For summaries, charts are the most prevalent type by far, with no papers using images or videos and only a few for each of the other types. Real-time feedback we found was in most cases of the sketch-like depictions type, with images or videos and other types being the least used ones. The exact statistics can be found in Table 3.3. As for the different feedback modalities, we believe that all types of visual feedback have the potential to be useful and effective depending on the specific use case and the users’ preferences. Many publications combined different types, such as bar charts with a cartoon dog [176]. We think such approaches are the most promising, as they can combine the benefits of multiple types, such as visual appeal and information density. While gamification elements are typically presented visually through scores, badges, or leaderboards, we see potential independent of the modalities of other provided feedback and think it and other additional methods should be considered. Looking at user studies conducted to evaluate visual sitting posture feedback, we found that out of 23 studies, seven included at least 24 participants, and nine were conducted over an extended period, i.e., more than one day. Only two had at least 24 participants and a run-time of more than one day. Most studies report positive results for various implementations of visual feedback, showing the general viability of such approaches. However, some studies revealed disadvantages or showed differences between feedback modalities or visualization types. One study found a decrease in productivity through the use of visual feedback [53] while flashing the computer display went unnoticed by the participants of another study [107]. Two studies compared visual feedback presented on a computer display with the use of physical objects, one finding the physical object to be less interrupting and disturbing [151]. The participants of the other study preferred the physical object and found it more visible and understandable while also expecting a better performance while using it Lee et al. [254]. Two studies comparing visual to vibrotactile feedback were not able to show significant differences in adoption compliance or cognitive load [255]. Two studies compared feedback with varying details, one reporting better results by providing more details [146], while the other could not find a significant improvement in the participants sitting posture even though they preferred more details [217]. 3.3 Summary and Findings Our literature review shows a large variety of approaches to sitting posture recognition and sitting posture feedback. Although pressure sensors are the most used hardware, we find that combining different types of sensors and other approaches like image processing are applicable as well. We found that there are multiple factors one has to consider when choosing an approach, like cost, privacy, portability, and accuracy. The second part of our review highlights that there is more research needed to better understand the differences between visual feedback and other modalities, as well as between various types of visual feedback. The overall positive influence of visual sitting 39 3 Literature Review about Sitting Posture Recognition and Feedback posture feedback should be encouraging to further the research in this area, as it gives a clear indication that such feedback can be effective. Even though this work focuses on visual feedback, we think that future research should not only consider various types and styles of visual feedback but also other modalities, such as sound or vibration. This way, users’ acceptance, which we presume to influence the effectiveness of feedback as well, is not limited by personal preferences or limitations. We further think that future studies should include a larger number of participants and run over extended periods of time to learn more about the long-term effects of sitting posture feedback. 40 4 Smart Chair Prototype Design This chapter details the hardware and software components we used to build our smart chair prototype for sitting posture detection and visual feedback, which can be seen in Figure 4.1. We first list, explain, and show the hardware components we used and describe their positioning on the chair, as well as the build process. Then we describe all the software components necessary for collecting, processing, storing, and visualizing the data. Finally, we present our three examples of visual feedback. 4.1 Hardware The base for our chair is an office chair by Interstuhl1, which we outfitted with four Force Sensitive Resistors (FSRs), three Time of Flight (ToF) sensors, and an ESP32 Microcontroller. We chose FSRs mainly due to their low profile, which makes them easy to install and unobtrusive, but also due to the lack of privacy concerns and their prevalence in related work. This reasoning is in line with Ordean et al. [259], who analyzed different methods for sitting postures with the Analytic Hierarchy Process (AHP) method. They found that one should find the user’s center of mass through FSRs and the upper-body tilt with ultrasonic sensors to optimize accuracy and avoid discomfort in a privacy-preserving way. We also chose to 3D print plates to increase the surface area of the FSRs, similar to Lamberti et al. [199], whose plates can be seen in Figure 4.2b. After initial tests, we introduced multiple laser-based Time of Flight (ToF) sensors at the backrest of the chair for better recognition of the user’s back posture. Our placement of both sensor types is similar to that of Li et al. [261], as shown in Figure 4.2a. We opted against ultrasonic distance sensors due to their comparatively larger build. Our goal is not to be able to differentiate a wide range of postures but, in concordance with current research, to support a user to frequently break up sitting time with postural shifts or low-intensity physical activity. For the pressure sensors in the seat, we used the FSRs model S15-450N2 produced by SingleTact. They have a circular sensing area with a diameter of 15mm and a weight-sensing range between 90g and 45kg. To be able to connect the FSRs to a microcontroller, we used the electronics board3 by SingleTact. An FSR attached to an electronics board can be seen in Figure 4.3b. We sawed off the sockets of the electronic boards to ensure the flattest possible installation. We attached the FSRs with double-sided tape to the wooden board below the cushion of the chair’s seat and added flat 3D-printed circular cones on top of them to increase their weight-detecting surface area. The cones have a diameter of 15mm at the bottom to match the size of the FSRs and a top diameter of 10cm, for which we were limited by the dimensions of the chair. 1https://www.interstuhl.com 2https://www.singletact.com/micro-force-sensor/standard-sensors/15mm-standard-sensor/15mm-450newton/ 3https://www.singletact.com/micro-force-sensor/singletact-electronics/electronics-board/ 41 https://www.interstuhl.com https://www.singletact.com/micro-force-sensor/standard-sensors/15mm-standard-sensor/15mm-450newton/ https://www.singletact.com/micro-force-sensor/singletact-electronics/electronics-board/ 4 Smart Chair Prototype Design (a) (b) Figure 4.1: Our smart chair prototype shown from the front, where the Time of Flight (ToF) sensors can be seen (a), and from the side with the enclosure of the microcontroller and the power bank visible in the lower right corner (b). One problem with these cones arose when we re-assembled the chair, as moving horizontally on the chair caused the cones, and thus the sensors, to be moved around. This resulted in one damaged sensor, one getting unplugged from the electronics board, and two being removed from its cone. We then tested the FSR model S15-45N4 with a weight-sensing range between 9g and 4.5kg without the cones to eliminate this problem. Unfortunately, even with the lower upper bound, they were not able to detect a person sitting on the chair without the cones, while they were too sensitive for use with the cones. We thus reverted back to the S15-450N model with cones but also added double-sided tape to the electronics boards and used electrical tape to further secure the FSRs and the cables. Figure 4.4a shows the pressure sensors and cones attached to the chair, while the exact measurements of the sensors’ positions can be seen in Figure A.1 in Appendix A. Our first approach included only the FSRs on the seat’s surface, as we assumed to be able to infer the user’s upper body posture from those values. As changes in posture, such as slouching, were not detectable through the pressure values, we eventually decided to add additional sensors for the user’s upper body posture. To measure the distance of the user’s back to the chair’s backrest, we used the laser-based Time of Flight (ToF) sensors model VL53L4CD produced by Adafruit5 with a 4https://www.singletact.com/micro-force-sensor/standard-sensors/15mm-standard-sensor/15mm-45newton/ 5https://www.adafruit.com/product/5396 42 https://www.singletact.com/micro-force-sensor/standard-sensors/15mm-standard-sensor/15mm-45newton/ https://www.adafruit.com/product/5396 4.2 Software (a) Li et al. [261] (b) Lamberti et al. [199] Figure 4.2: Examples of inspiration for our sensor placement (a) and 3D printed plates to increase the area of the pressure sensors (b). detectable range between 1 and 1300 millimeters, shown in Figure 4.3a. We removed some of the chair’s foam and drilled holes through the wooden backrest to attach the sensors from behind to ensure they do not cause discomfort to the user and also reduce the chance of someone accidentally damaging them, as they are very sensitive. Figure 4.4b shows the backrest of the chair from behind with the attached ToF sensors, while their exact positions viewed from the front can be seen in Figure A.2 in Appendix A. We connected all sensors to a NodeMCU ESP32 microcontroller6, as shown in Figure 4.5, and powered it through a 5000mAh power bank. The ESP32s built-in Analog to Digital Converter (ADC) converts the analog voltages of the FSRs, while we use its Inter-Integrated Circuit (I2C) interface to read the output of the ToF sensors. The wiring was done on a stripboard that we screwed into an aluminum box for electronic projects, out of which we guided the wires through holes. We used Velcro strips to secure the box to the chair’s underside and to attach the power bank to the box. To access the values collected by the ESP32, we wrote a python module that wirelessly connects to it through Bluetooth and receives the data. 4.2 Software In this section, all the software components we developed for our smart chair prototype are presented. First, we describe the collection of sensor data on the microcontroller. Then, our python module SmartChair and its functions, from receiving the data to presenting visual feedback, are described in detail. All of our