Institute for Visualization and Interactive Systems
University of Stuttgart
Universitätsstraße 38
D–70569 Stuttgart
Bachelorarbeit Nr. 317
Correlation between Measured
Stress and Attention with
Thermal Imaging Techniques
Thommy Zelenik
Course of Study: Softwaretechnik
Examiner: Dr. Niels Henze
Supervisor: Dipl.-Medieninf. Tilman Dingler,
Yomna Abdelrahman, M.Sc.
Commenced: 5. April 2016
Completed: 5. October 2016
CR-Classification: I.7.2

Abstract
Thermal imaging techniques enable an easier way to detect interaction compared to RGB
or depth cameras and are therefore more comprehensively used in Human Computer
Interaction (HCI) as a sensory system enabling novel interactive systems with the
help of computer vision techniques. This makes it possible to use a thermal camera
which operates in the Far Infrared (FIR) spectrum for the processing of humans skin
temperature which is indicative of stress and therefore suits for detecting stress.
Stress is considered of being one major cause for several sicknesses. Human’s health
can suffer because of stress and this includes also one’s performance. Stress can effect
the attention to different tasks. The evolution of techniques especially the computer
technology overwhelmed a lot of people with an increasing need for computers in
every department of work. This enables multi-tasking with several tasks being done
simultaneously quite easily but also enabling a surge of stress. Having several tasks
simultaneously to do can affect the outcome of them because a lack of concentration can
occur or attention to the important task drops because of stress. The correlation between
stress and attention can be investigated and based on a positive relation between them
one can point to stress decreasing mechanisms in the case of performance slumps.
In this thesis we show that through thermal imaging techniques it is possible to detect
objects of interest. We investigate how thermal cameras can be used for detecting
one’s stress and which areas of a human are suitable for measuring stress. We moreover
discuss the relation between stress and attention. Based on this researches an application
will be implemented which will be able to detect stress. Using this application in an user
study, where subjects perform several different stress generating tasks, we examine how
stress affects attention and vice versa. Furthermore some areas which could benefit from
the examined stress and attention relation are named.
Keywords: Thermal imaging, Stress detection, Stress attention correlation, human
detection, Stress detection application
3

Abstract
Wärmebildkameras machen es möglich Interaktionen einfacher als RGB Kameras zu
erkennen und werden daher in der Mensch Computer Interaktion (HCI) als ein Sen-
sorsystem benutzt, wodurch neuartige interaktive Systeme mit Hilfe von Computer
Vision Techniken möglich werden. Dies macht es möglich eine Wärmebildkamera, die im
fernen Infrarot Spektrum operiert, zu benutzen um die Hauttemperatur von Menschen
zu verarbeiten, die hinweisend für Stress ist und daher gut zur Stresserkennung dient.
Stress wird als eine der Hauptursachen für viele Krankheiten angesehen wobei dadurch
die Gesundheit und Leistungsfähigkeit eines Menschen leiden kann. Stress kann die
Aufmerksamkeit auf verschiedene Aufgaben beeinflussen. Die Technologieentwicklung,
vor allem die der Computertechnologie, überforderte viele Menschen mit einem größeren
Computerbedarf in allen Arbeitsbereichen. Das machte Multitasking auch am Arbeit-
splatz leichter möglich. Diese Arbeitsweise kann aber durch einen Konzentrations- oder
Aufmerksamkeitsverlust aufgrund von Stress das Ergebnis beeinflussen. Die Beziehung
zwischen Stress und Aufmerksamkeit kann untersucht werden und bei einem posi-
tiven Ergebnis stressreduzierende Maßnahmen vorgeschlagen werden falls sich ein
Leistungsabfall bemerkbar macht.
In dieser Abschlussarbeit werden wir zeigen, dass es durch Wärmebildtechniken möglich
ist gewünschte Objekte erkennen zu lassen. Wir werden untersuchen wie Wärmebild-
kameras benutzt werden können um Stress zu erkennen und welche Körperteile eines
Menschen dafür geeignet sind. Darüber hinaus werden wir die Beziehung zwischen
Stress und Aufmerksamkeit untersuchen. Basierend auf die Untersuchungen werden
wir eine passende Applikation programmieren, die Stress erkennen kann. Diese App-
likation wird in einer Nutzerstudie eingesetzt, in der die Teilnehmer viele verschiedene
stresserzeugende Aufgaben bearbeiten um zu untersuchen wie Stress und Aufmerk-
samkeit zusammenhängen. Des Weiteren werden einige Anwendungsgebiete aufgezählt
welche von den Ergebnissen profitieren können.
Schlüsselbegriffe: Wärmebild, Stresserkennung, Beziehung zwischen Stress und
Aufmerksamkeit, Menscherkennung, Stresserkennungsapplikation
5

Contents
1. Introduction 11
1.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.2. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2. Background 15
2.1. Thermal Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2. Ways of interacting with humans using thermal imaging techniques . . . 16
2.3. Stress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4. Attention and stress relation . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.5. Kind of stressors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.6. Implementation Techniques . . . . . . . . . . . . . . . . . . . . . . . . . 22
3. Related Work 29
3.1. Measuring stress in the prefrontal and periorbital facial area . . . . . . . 29
3.2. Masuring stress using nasal skin temperature . . . . . . . . . . . . . . . 38
4. Algorithm 41
4.1. Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.2. Implementation in C# . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.3. Communication to the process imager device . . . . . . . . . . . . . . . 42
4.4. Data representation of the thermal image . . . . . . . . . . . . . . . . . 43
4.5. Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5. User Study 49
5.1. Region of interest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5.2. Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5.3. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
6. Summary and outlook 65
6.1. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.2. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6.3. Potential field of application . . . . . . . . . . . . . . . . . . . . . . . . . 67
7
A. Glossar 69
A.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Bibliography 75
8
List of Figures
1.1. Electromagnetic spectrum . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1. The Composition of IR radiation . . . . . . . . . . . . . . . . . . . . . . . 15
2.2. Main principle of non-contact thermometry . . . . . . . . . . . . . . . . 17
2.3. OpenCV Main Components . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4. Haar-like input features . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.5. Using the Haar-like input features on example faces . . . . . . . . . . . . 26
2.6. Decision tree with three nodes . . . . . . . . . . . . . . . . . . . . . . . . 27
3.1. False colour image of a subject before and after a startle . . . . . . . . . 30
3.2. Thermal image of subjects before and after emotional stressors are applied 31
3.3. Image of a subject who underwent an emotional and phyiscal stress . . . 32
3.4. Thermal immage showing hot pixel patterns due to emotional stress and
high fever . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.5. Thermal pictures of a subject undergoing a running exercise as a physical
stressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.6. Thermal image of a subject with the region of interest being the forehead
region . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.7. Stress detection using nasal skin temperature slope . . . . . . . . . . . . 39
3.8. Nasal area temperature change during the experiment . . . . . . . . . . 40
4.1. Measurement field of the infrared camera optris PI . . . . . . . . . . . . 42
4.2. The basis implementation which gets and displays the temperature . . . 43
4.3. Method for getting the temperature of a pixel . . . . . . . . . . . . . . . 43
4.4. The Imager.exe software for getting the thermal image data . . . . . . . 45
4.5. CascadeClassifier holding the trained data from the XML file . . . . . . . 45
4.6. Image of the Facedetection algorithm . . . . . . . . . . . . . . . . . . . . 46
4.7. Image of the final application . . . . . . . . . . . . . . . . . . . . . . . . 47
5.1. The descriptive statistics results of the "scary" and "info" tasks of the
"Videos" condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.2. The descriptive statistics results of the "boring" and "entertaining" tasks
of the "Videos" condition . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
9
List of Figures
5.3. The descriptive statistics results of the "comic" and "easy blog" tasks of
the "Reading Text" condition . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.4. The descriptive statistics results of the "literary piece" and "science article"
tasks of the "Reading Text" condition . . . . . . . . . . . . . . . . . . . . 54
5.5. The descriptive statistics results of the "level 1" and "level 2" tasks of the
"Stroop color game" condition . . . . . . . . . . . . . . . . . . . . . . . . 55
5.6. The descriptive statistics results of the "level 3" and "level 4" tasks of the
"Stroop color game" condition . . . . . . . . . . . . . . . . . . . . . . . . 55
5.7. Summary of the ANOVA for all conditions . . . . . . . . . . . . . . . . . 56
5.8. The ANOVA for all conditions . . . . . . . . . . . . . . . . . . . . . . . . 56
5.9. Summary of the ANOVA for all tasks within each condition . . . . . . . . 57
5.10.The ANOVA for all tasks within each condition . . . . . . . . . . . . . . . 57
5.11.The order of the most stress and attention generating tasks . . . . . . . . 58
5.12.Participant 3: Started in a relaxed state playing level 3 observing emerging
stress as the game prolonged . . . . . . . . . . . . . . . . . . . . . . . . 60
5.13.Participant 3: Remaining in this stressful state while playing level 2 seeing
the temperature gap getting bigger . . . . . . . . . . . . . . . . . . . . . 60
5.14.Participant 3: Seeing the participant getting irritated because of the
performance loss while playing level 1, only being able to get low scores
on the easiest difficulty . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.15.Participant 3: Achieving even bigger stress while playing level 4 see-
ing the participant being stressed out observing his lowest nasal area
temperatures in this experiment . . . . . . . . . . . . . . . . . . . . . . . 61
10
1. Introduction
Scientists have always been fascinated by diagnostic temperature phenomena not only
because of today’s technological advancements for remote temperature measurements
but since Hippocrates’ early understanding to Galileo’s famous thermoscope [Rin04].
These current technological advancements allowed measurement of emitted infrared
heat by electronic thermal imaging [Rin04] [RA12] which shows how this technology
developed with Sir William Herschel being the first scientist who measured heat beyond
the visible spectrum in 1800 [Rin00] and producing the first thermogram in 1840 using
sunlight and the evaporograph technique. Infrared imaging provides versatility with
enabling the recording of perspiration [Ebi+12] [Pav+12], blood flow [Pur+05a], cuta-
neous and subcutaneous temperature variations [Hah+12] and cardiac pulse [Gar+07]
and also enables one to infer psychophysiological excitement but also differentiates
between baseline and affective states [NC10a].
Body temperature is of particular significance to medicine because homeostatic control
of cutaneous temperature is functional for both biological and psychological reasons like
facing an environmental change, fighting a virus [Ski+07] or supporting physiological
demands in case of an external threat [Por01a] and therefore giving a lot of information
about ones health since there is a correlation between body temperature and diseases
with in the beginning using from today’s point of view unhandy and inconveniant ther-
mocouples, thermistors and thermopiles to record skin temperature [Rin90] [AGH98].
The thermal imaging technique is a non-contact, noninvasive, fast, reliable and safe
method for patients and doctors allowing multiple recordings at short time intervals
[Hsi+90]. It includes medical imaging and condition monitoring in industries by measur-
ing the surface temperature of a body or an object [Law56]. Thermal imaging techniques
are already used for studying blood flow which can be assessed with different techniques
and one of them being medical infrared thermal imaging [Jon98].
Certain diseases have been detected by observing changes in the skin temperature using
thermal images [DC95][CTN05].
Pathological processes in the humans organs manifest themselves as local changes in
heat production and also as changes in the blood flow pattern of the affected organs or
tissues [ZLA04].
Infrared imaging is already heavily used in clinical diagnostics as a physiological test that
measures the subtle physiological changes that might be caused by mandy conditions
with them being commonly associated with different diseases which generate a higher
11
1. Introduction
Figure 1.1.: The Electromagnetic spectrum [Lar+11]
temperature heat source [Vai05] [Vai00].
One of them being stress which can lead to several diseases [CJDM07].
The thermal imaging technique hasn’t been used that much in today’s world since ther-
mal cameras always have been expensive but getting cheaper and therefore affordable
as time passes and better technology is being developed. Thermal cameras operate in a
specific band which can be shown in the electromagnetic spectrum as shown in Figure
1.1.
It is divided into several bands with the RGB band being the most explored. Thermal
cameras are used for visible imaging since there are several libraries for interaction in
this light spectrum band. It features color and illumination robustness and therefore be-
comes more interesting in HCI and especially for research groups since thermal imaging
techniques are not only used in military and medical departments anymore [Ziv04].
1.1. Motivation
Physiological changes like increased body temperature at certain areas can give some
indication of stress. The technique for doing so is thermal imaging. This includes the
definition of the key areas on the body which are ideal for the observation of temperature
changes and therefore suited for the observation of stress level changes.
The research areas include stress and attention correlation, particularly the amount one
pays attention to someone else when under stress but also if it is possible to measure
the attention level via thermal imaging techniques in the same way as when measuring
stress levels. This includes the question if the facial regions for measuring this are the
same.
In order to investigate this an application is needed which detects the regions of interest.
Also it should be able to get and process the temperatures. So a thermal image camera
will be needed for the detection of temperatures in the facial region.
12
1.2. Overview
1.2. Overview
The structure of this work is as follows:
Chapter 2 – Background: The work starts with some background information which
encompasses the basic definition and description of thermal imaging followed by a
short preamble of the different ways of interacting with humans using a thermal
camera. Then the two terms stress and attention are explained in big detail which
is followed by an overview of the two kinds of stressors and is being completed by
the implementation techniques used for this thesis.
Chapter 3 – Related Work This chapter discusses the previous work using thermal
imaging for stress and attention detection focusing on work where the region of
interests have been the forehead and the nasal area.
Chapter 4 – Algorithm This chapter deals with the implementation in great detail with
a introduction to the hardware, being the thermal camera, and the interactive
prototype that exploits the introduced concepts is described here.
Chapter 5 – User Study This contains the user study where at first the regions of
interest are determined followed by the experiment protocol and is completed by
the results.
Chapter 6 – Summary and outlook This chapter provides a summary and an outlook
for future research.
13

2. Background
In the following some background information which encompasses the basic definition
and description of thermal imaging followed by a short preamble of the different ways
of interacting with humans using a thermal camera is provided. Then the two terms
stress and attention are explained in big detail which is followed by an overview of the
two kind of stressors and is being completed by the implementation techniques used for
this thesis.
2.1. Thermal Imaging
The change in intensity of the radiation depends on the temperature changes of the
target because every body sends out a predetermined amount of radiation dependent
on its temperature. The infrared radiation was discovered in 1800 by William Herschel
during a search of new optical materials blackening the peak of a sensitive mercury
thermometer. He tested the heating of different colors of the spectrum by using a
thermometer, a glass prism that led sun rays onto a table making his measuring arrange-
ment. He found the maximum temperature far behind the red area being called infrared
wavelength area by slowly moving the peak of the blackened thermometer through the
Figure 2.1.: The Composition of IR radiation [Gmb]
15
2. Background
colors of the spectrum noticing the increasing temperature from violet to red.
Infrared thermometry uses a wave-length ranging between 1 micrometer and 20 mi-
crometers for the measurement of "thermal radiation" with the intensity of the emitted
radiation depending on the material. Infrared thermometers calculate the surface tem-
perature contactless on the basis of the emitted infrared radiation from an object [Gmb].
Figure 2.1 shows that the surface material of the measured object has radiation-features
which are vital for the intensity of infrared radiation and it depends also on the tem-
perature. The formula is E + p + T = 1 where E represents the emissivity used as a
material constant factor to describe the ability of the body to emit infrared energy and
can range between 0 and 100%. A bad radiation source would be a mirror which shows
an emissivity of 0.1 whereas a "blackbody" would be the ideal radiation source with an
emissivity of 1.0 [Gmb].
2.2. Ways of interacting with humans using thermal
imaging techniques
2.2.1. Contact
To measure physiological data so called stress markers can be assessed. These are heart
rate, heart rate variability, finger temperature, the salivary alpha-amylase and cortisol
[Eng+14].
2.2.2. Contactless
A contactless method for capturing physiological data is the usage of thermal imaging
cameras. It is suitable for covert recordings or to focus on special populations which
show difficulties in complying with the standard instruments of data collection. It is
also beneficial in the domain of psychophysiological covariance research [Eng+14].
Infrared thermometers basically consist of the components lens, spectral filter, detector
and several electronics like a amplifier, linearization or signal processing to be seen in
Figure 2.2.
A lens needs to be specified as it decisively determines the optical path of the infrared
thermometer characterized by the ratio distance to spot size. The wavelength range
relevant for the temperature measurement is selected by the spectral filter and the
emitted infrared radiation is being transformed into electrical signals by the detector in
cooperation with the processing electronics [Gmb].
16
2.3. Stress
Figure 2.2.: Main principle of non-contact thermometry [Gmb]
2.3. Stress
Humans want to do sometimes too much in a time period which offers less time for all
the things that are planned. Hereafter acute stress will be described and also distress
which is a negative form of stress, because in the end there will be an explanation how
stress can cause positive changes in life being called eustress [Sel76].
The industrial society demands a busy and industrial life which can lead to worries for
the future and leaves little time for family und hobbies but a stressless life would not
offer any challenges between one’s needs and the extern environment and therefore
no evolution or survival would exist, no ways for improving oneself. An example for a
stressor is moving up the deadline for the delivery of a complementary work causing one
to work faster and being more productive reducing time for other things leading to a
reaction which disturbs one’s balance and strongly demands or surpasses one’s abilities
for overcoming it.
Individuals react to dangers on physiological, behavioral, emotional and cognitive levels.
A physiological reaction would be a sleeping disorder because of always thinking about
the stressful situation. A behavioral reaction would be doing overtime in order to meet
the deadline of a complementary work. An emotional reaction would be seeing one’s
anger because one isn’t able to control his anger anymore. And a cognitive reaction
would be having a hard time concentrating on the work [GZ08].
2.3.1. Chronical stress
Chronic stress is an enduring state of arousal which lasts a long period of time where
the inner and outer resources do not seem to be enough for overcoming this stress like
17
2. Background
seeing the impossibility of finding enough time for doing all the things desired [KRC81].
Chronical stress is most likely experienced by humans with a weak socioecological state
or by humans of suppressed races [Tro+03] [Sto00] [LB97].
2.3.2. Stress physiology
Walter Cannon described in the 1920’s that humans and animals react to dangers with
the activation of nerves and glands causing the fight or flight reaction meaning the
defending and fighting or to flee to secure safety. However it is argued that only men
have this fight or flight reaction and women show signs of welfare and contentment
when under stress [Tay+00].
The physiological procedures start with the hypothalamus acting as the stress centre
performing the emotional reactions of governing the autonomous nerve system and
activating the hypophysis. The autonomous nerve system regulates the body organ
activities with common body reactions to stress being faster breathing and heart rate,
tightened blood vessels and an increased blood pressure. Muscles open the way between
pharynx and nose for better breathing and muscles also direct emotional face expressions.
Messages are being sent to smooth muscles for preparing digestion and other body
functions to be blocked because they are considered to be irrelevant in emergency
situations. The inner part of the adrenal glands, the adrenal medulla releases adrenalin
and noradrenalin making organs like the splenic releasing red blood cells, the bone
marrow producing more white blood cells and the liver producing more sugar as energy
supplier to the body.
The hypophysis releases the important stress reaction hormones thyroidea stimulating
hormone (TSH) which makes the thyroid releasing more energy for the body, and the
adrenocorticotropic hormone (ACTH) known as the "stress hormone" making the adrenal
cortex releasing hormones for controlling circulatory processes and releasing sugar from
the liver to the blood. ACTH makes several organs releasing thirty other hormones for
an emergency reaction [Sel50].
2.3.3. General adaption syndrome
The reactions on stress situations are described as the "general adaption syndrome"
which has three levels. The first one being the alert reaction which has the characteristic
of having short periods of physical arousal which prepares the individual for vigorous
actions. If the stressor persists the body enters the second level of resistance which
is a state of moderate arousal where the organism has the ability of defending itself
from weakening effects of the persistent stressor. If the stressor lasts long enough the
18
2.4. Attention and stress relation
individual enters the last level which is the state of exhaustion where body ressources
are tiring out [Sel56] [Sel13].
2.4. Attention and stress relation
Paying attention to someone or something means to choose to concentrate on a discrete
aspect of information and ignoring other perceivable information. It is a major area in
cognitive psychology with investigations regarding the physiology and the relationship
between attention and other behavioral and cognitive processes like working memory
and vigilance.
The question is how stress and attention are correlated and the conclusion is that our
ability to process information is limited and under stressful conditions, the cognitive sys-
tem becomes overloaded, decreasing a person’s attentional resources. As stress increases
and attention becomes more selective, there is a growing exclusion of information that
is irrelevant to the task at hand suggesting it can be beneficial when a task requires
the exclusive focusing on target information. On the contrary stress can also lead to
increased distractibility of the individual because the reduction of attentional resources
under stress may result in a decreased ability to filter out irrelevant information from
relevant information. But there is evidence that tasks that require the integration of
information from several sources are vulnerable to the effects of stress. So under stress
one will concentrate on one target when being confronted with multiple targets leading
to impaired performance, excluding other relevant targets. Also a extremely narrow per-
spective and also tending to stop considering other possible diagnoses after a diagnosis
is reached are likely to occur [LeB09].
So in general under stress attention appears to channel or tunnel, reducing focus on
peripheral information which depends on the perception of each individual to be of
greatest importance to the individual and reducing focus on tasks and centralizing focus
on main tasks. Tunneling of attention can result in either enhanced performance or
reduced performance, depending on the nature of the task and the situation. Like when
peripheral cues are irrelevant to task completion the ability to turn them out is likely to
improve performance and when these peripheral cues are related to the task and their
incorporation would otherwise facilitate success on the task, performance suffers when
they are unattended [Sta04].
19
2. Background
2.5. Kind of stressors
Facial muscles are controlled by the brainstem through the myelinated vagus causing
expressions according to psychological and environmental factors and like all organs
of the body requiring nutrients which are supplied through an adjustment in the blood
stream to cover muscular activity changing the emitted thermal print [NC10b].
Facial expressions have been argued to be a behavioral gateway for healthy psychological
functions like autism, aggressive disorders or schizophrenia and constitute an integral
part of interpersonal socioemotional signaling [Por01b].
Supraorbital [Pur+05b] and periorbital vessels [LPC01] of the face have been observed
to show heat escalations according to stressors that are believed to facilitate preparedness
for rapid eye movement in fight of flight [PLB01] with supraorbital regions having been
postulated to represent prolonged periods of stress due to mental engagement [ZTP07]
[Pur+05b] and the periorbital region has been suggested to carry information about
short-lived stressors such as startles [LPC01] [Nak+05] which are controlled by the
midbrain central gray matter and the nucleus of the tractus solitaris in the pons [ZD04].
Otherwise startles trigger a temperature dip in the cheeks suggesting to be a result
of redirected blood to the eye musculature as well as of emotional sweating [MR07]
feeding the main muscles surrounding the eyes being the corrugator, procerus and
orbicularis oculi.
Stress is a physical or physiological imbalance which one sees as a threat. Physical
stressors are considered as an external condition because of heat, cold or noise or as an
internal demand of the human body. An emotional stressor has no direct impact on the
human body and can affect the cognitive or the emotional system in the brain. Emotional
stressors may place demands on either the cognitive systems (thought processes) or
the emotional system (feeling responses, such as anger or fear) in the brain but such
body responses to these two types of physical and emotional stressors can be trained to
the extent that the tolerance to them increases leading to slightly different reactions to
them.
A physical response to stress can be monitored by evaluating the blood volume during
stress. The observation leads to the conclusion that the brain causes different reactions
to stress like:
• an increase of blood pressure and oxygen transport by the blood throughout the
body causing more demand of the muscles, lungs and brain because of a huge
blood flow increase
• stickier blood
• acceleration of heart and lung action
• dilation of pupils
20
2.5. Kind of stressors
There was a 100% increase of these physiological responses that were evaluated
through an experiment where controlled dosages of epinephrine (adrenaline) and
norephinephrine (noradrenaline) were injected into the bodies of dogs [CG80].
Other physiological responses to stress are:
• liberation of nutrients for muscular action like glucose and oxygenation
• Skin paling or flushing or alternating between both
• constriction of blood vessels in several parts of the body like skin, stomach and
intestine
• more instantaneous reflexes and sweat
• rational thought, navigation ability, short term memory and concentration are
suppressed
• effect on the sphincters of the body, tunnel vision and loss of hearing
• hindered tear production, digestion and erection and mouth dryness
• relaxation of bladder and evacuation of colon
[Hon+09]
21
2. Background
2.6. Implementation Techniques
In the following the most important implementation techniques are introduced especially
providing some basic knowledge about face detection with EmguCV.
2.6.1. Introduction to C#
C# is an object oriented language which enables the creation of applications which
run on the .NET framework. It was developed by Microsoft within its .NET initiative
with Anders Hejlsberg forming a team in january 1999 to build a new programming
language named Cool which meant "C-like Object Oriented Language". The language
had been renamed to C# in July 2000 where it was publicly presented at the Professional
Developers Conference and also the class libraries and the ASP.NET runtime had been
ported to C#. C# has the following features, enabling:
• secure and robust applications
• windows client applications
• XML Web Services
• distributed components
• client-server applications
• database applications
• type-safe event notifications through encapsulated method signatures called "dele-
gates"
• private member variables which can be accessed through properties
• declarative metadata about types at run time which are provided through attributes
• inline XML documentation comments
• built-in query capabilities across a variety of data sources through the Language-
Integrated Query (LINQ)
[Mic03] [Mic15b]
22
2.6. Implementation Techniques
2.6.2. .NET Framework
The common language infrastructure (CLI) is an international standard where languages
and libraries can work together in order to create or develop applications. Microsoft
used this standard for the commercial creation of their virtual execution system called
common language runtime (CLR) which in combination with a set of class libraries
forms the .NET Framework for the execution of C# applications [Mic15c].
2.6.3. Microsoft Visual Studio 2015 Community
Visual Studio as an integrated development environment supports C# programs too and
therefore has been considered as perfect for this work. The development of software in
general is supported well with tools for UI design, coding, testing, debugging, analyzing
code quality and performance, deploying to customers and gathering telemetry on usage
[Mic15a].
2.6.4. OpenCV
In order to realise an application which is able to detect temperature increases in special
areas a library was needed which facilitates this intention and OpenCV seemed as
the perfect solution for it since developers hardly start from scratch in defining every
operation and an image processing library improves the efficiency [Ope08].
2.6.5. EmguCV
Since OpenCV supports C and C++ and the preferable programming language was
C# a .Net wrapper to the OpenCV image-processing library was needed. EmguCV is a
cross-platform image-processing library making it possible to directly call functions or
methods written in native C or C++ [Shi13].
23
2. Background
Figure 2.3.: OpenCV Main Components [Shi13]
Wrapping OpenCV
In order to understand the wrapping process it is vital to know that OpenCV is divided
into the components pictured in Figure 2.3.
The CV component contains the image processing and vision algorithms, the machine
learning(MLL) algorithms contains clustering tools and statistical classifiers and the
HighGUI needed for user-friendly interfaces loads and stores media data.
CXCORE contains all basic data structures and contents and is completed by CVAux
which has on the one side defunct areas and on the other side experimental algorithms
not to be seen in Figure 2.3.
EmguCV consists of two layers with Layer 1 being the basic layer where the namespaces
are direct wrappers from OpenCV components having
• enumerations
• structures
• function mappings
and Layer 2 taking advantage of the .NET framework and mixing the classes together.
There are several DLLs which are used the following way:
• Emgu.Util.dll which is a collection of .NET utilities
• Emgu.CV.dll which are the basic image-processing algorithms from OpenCV
24
2.6. Implementation Techniques
• Emgu.CV.UI.dll which contains useful tools for Emgu controls
• Emgu.CV.GPU.dll used for GPU processing (Nvidia Cuda)
• Emgu.CV.ML.dll having machine learning algorithms
[Shi13]
25
2. Background
Figure 2.4.: Haar-like input features [Shi13]
Face Detection
The interesting part was the face detection which is regarded as a self-contained applica-
tion and was developed earlier than other Machine Learning functions.
Biometric systems are developed to automatically verify a person from input data like
infrared data which is a little bit problematic since face detection is a problem in biomet-
rics and biometric systems have disadvantages since they are not nonintrusive unlike
face detection where face images can be captured in a covert manner.
Machine-learning algorithms allow face detection and the Emgu.CV.ML namespace
includes the CascadeClassifier class with its algorithm "The Cascade classifier".
Paul Viola and Michael Jones implemented the first face detection technique called
"HaarCascade" with "Haar" meaning the "Haar" features which calculate rectangular
image regions and threshold the result to train them into cascade files. There exist
already pretrained ones, one of them being the default XML file for front-face detection.
The Viola-Jones classifier uses Haar-like input features, a threshold applied to sums and
differences of rectangular image regions as shown in Figure 2.4 where light regions are
interpreted as "add this area" and the dark region as "subtract this area" applying this
rules leads to the more meaningful Figure 2.5.
After using the integral image technique to accelerate computation the next step is to
create binary classification nodes of a decision tree like in Figure 2.6 where after all
the Haar features calculation, the image that passes all the nodes is regarded as a face
image [Shi13].
26
2.6. Implementation Techniques
Figure 2.5.: Rule that the color of human eye area looks deeper than the cheek area
placing this feature in the adjacent rectangle of the eyes and cheek area
using this for face detection [Shi13]
Figure 2.6.: Non-leaves are judgements, paths are the result of the last judgement and
each leaf is a kind of output whether being face or not [Shi13]
27

3. Related Work
In order to implement the application which detects stress using a thermal imaging
technique some important previous work is needed. This chapter provides an overview
of several papers which used different technologies and conducted different experiments
which are all described in detail. Some of these works also deal with stress detection but
different sensors and processing methods were used however providing vital knowledge
about the topic. Some important related works were separated into "The experiment"
and "Results" to describe the process in more detail.
3.1. Measuring stress in the prefrontal and periorbital facial
area
There are many great works which deal with the measurement of stress using thermal
imaging techniques figuring out that the prefrontal and periorbial facial area suit this
purpose. In the following some of them are presented.
3.1.1. "The face of fear" by Ioannis Pavlidis et al.
Ioannis Pavlidis has a lot of works relating to this topic breaking the first ground with the
thermal imaging work "The face of fear" where six participants underwent an experiment
with different stressors to investigate how regional, facial blood flow changes with fright
all captured by a thermal camera. One result was a warming over the participant’s
mandibular region when chewing gum.
Another one was a periorbital warming, cheek cooling and an increase in blood flow
to eye musculature caused by a sudden loud noise as a startle as shown in Figure 3.1
[LPC01].
29
3. Related Work
Figure 3.1.: The temperature around the periorbital region of the subject increased by
almost 1C after the startle [LPC01]
3.1.2. “Detection and classification of stress using thermal imaging
technique" by Kan Hong et al.
In the work "Detection and classification of stress using thermal imaging technique" they
found out that other areas like the forehead, neck and cheek also react with increased
skin temperature triggered by stress and that the forming pattern depend on whether it
is a physical or an emotional stressor. For the observation of the experiments a thermal
imaging technique was used for monitoring the blood volume by evaluating the skin
temperature in the facial region and to detect stress by using a hyperspectral imaging
(HSI) technique for detection of a change of the haemoglobin oxygenation level (HOL).
All participants had to endure a series of emotional and physical stressors having been
watched by heart monitors during the experiments.
The experiment
The setup consisted of participants sitting in an arm chair with heated black boxes for
temperature calibration and mirrors for reflecting light in the background. The cameras
have been put 3-4 meters away from the participants and with the help of broad band
halogen lamps the participant sat in bright light. The thermal images based on a 30hz
frame rate.
The experiment started with giving the participants a quiz and mental arithmetic as
30
3.1. Measuring stress in the prefrontal and periorbital facial area
Figure 3.2.: Emotional stressor triggers an over 50 percent increase of hot pixels
pictured as green spots in the forehead, eardrum, lip and neck regions
[Hon+09]
emotional stressors. The observation was an increase of heart beat from 75 BPM (beat
per minute) to 100 BPM. They figured out that their participants, even when they are
calm and relaxed at the start of the experiment, had patches of hot skin temperature in
the periorbital, prefrontal, ear and neck regions. The conclusion was that the thermal
patterns depend on the health and mood of each participant. They compared their
solution with the one from Pavlidis and saw that over 90% of increased hot pixel counts
were found in the forehead, around the lip and ear drums but very little change of hot
pixel counts in the periorbital region in great contrast to the results from Pavlidis as
shown in Figure 3.2.
The cause could be the sort of the stressor, theirs being an emotional stressor and in
Pavlidis’s experiment the startle stressor was a physical one.
This was their first observation and reasoning. To underline this theory a further
experiment with a running exercise as another physical stressor has been done. The
participant sat in the beginning comfortably on a chair for a few minutes and was then
asked to run vigorously upstairs for a couple of times and to return after two minutes.
In that resting time another set of thermal images was taken.
The difference in the thermal images is shown in Figure 3.3 with the reaction to the
emotional stressor being on the left side and the reaction to the physical stressor being
on the right side.
31
3. Related Work
Figure 3.3.: There are two very distinct hot pixel patterns that have been induced by the
two different types of stressors (left emotional and right physical stressor)
[Hon+09]
Figure 3.4.: High fevers induce big patches of hot pixels mainly at the centre of the
prefrontal and periorbital region different from the one due to emotional
stress [Hon+09]
Results
This result assured Pavlidis’s findings. The physical stressor leads to more hot pixels in
the periorbital region and less in the forehead while the emotional stressor caused more
hot pixels in the prefrontal region. In order to understand the physiological response
to the physical stressor another physical stressor named "The Brain Stem Activational
Test" has been applied. The procedure guarantees that after a resting time of 30 minutes
the participant breathes in large doses of CO2 for a few seconds and is pertained as
a different kind of physical stressor because the participant has to sit throughout the
session and the CO2 triggers the brain stem to feel breathless. The result of this process
is similar to the previous one having a more complex thermal pattern in the prefrontal
32
3.1. Measuring stress in the prefrontal and periorbital facial area
region by the emotional stressor than that of the two physical stressors. The two physical
stressors caused an increase of hot pixel counts in the periorbital region and less in the
prefrontal region. Another differentiation of hot pixel patterns have to be made when
considering high fever. High fevers induce big patches of hot pixels mainly at the centre
of the prefrontal region with elevated temperatures in the periorbital areas which are
different from the one due to emotional stress as shown in Figure 3.4 [Hon+09].
3.1.3. “Thermal Image Analysis for Polygraph Testing” by Ioannis
Pavlidis and James Levine
Pavlidis was also involved in the work "Thermal Image Analysis for Polygraph Testing"
where additional to the polygraph testing infrared facial image analysis was used despite
an average polygraph testing success rate of 90% caused by the assessment of the
physiological parameters blood volume and pulse changes, respiratory changes and
electrodermal activity. Infrared facial image analysis was used because it is noninvasive
which benefits the participant’s comfort and makes it possible to get physiological infor-
mation similar to the ones of polygraph testing like the blood flow rate.
While performing the polygraph test they could observe only unnoticeable temperature
changes via thermal imaging which was in stark contrast to the previous startle experi-
ment described in "The face of fear" justifying it with the stress being subtle compared
to the startle one. Another finding is the increased blood flow circulation around the
eyes being associated with anxious states.
They realised a detailed visual representation of facial blood flow rate patterns and
corresponding eye curve slopes by monitoring the temperature changes in the periorbital
regions of a subject who was programmed nondeceptive in order to pass the polygraph
test and a second subject who was programmed deceptive to fail the polygraph test.
There was a strong disparity of the eye slopes in the corresponding answer sessions
leading to the conclusion that steep eye curves are indicative of a deceptive answer.
This result however was not that definite as the interrogation scenario was not a normal
one. The restrictions were several with asking questions to the subjects multiple times,
having long pauses between each question or the subjects being bounded to giving binary
answers "yes" or "no" and instructing the subjects to stay as still as possible [PL02].
3.1.4. “Imaging facial physiology for the detection of deceit” by Ioannis
Pavlidis et al.
To adjust the restrictions to a natural interrogation scenario with continuous flow and to
build up psychological pressure by accelerating or decelerating the interrogation pace
33
3. Related Work
another paper with Pavlidis’s involvement is analysed entitled "Imaging facial physiology
for the detection of deceit". Also the quality of the input thermal frames is improved by
using a noise reduction algorithm as a filter and then a pattern recognition algorithm is
applied to decide whether the answer was deceptive or non-deceptive.
Their results encompassed the periorbital region as the region of interest to be more
clearly defined as the inner corner of the eyes where there is a supply of blood flow to
the eye musculature [Tsi+07].
3.1.5. “Interacting with human physiology” by Ioannis Pavlidis et al.
Another related work of Pavlidis is "Interacting with human physiology" where he was
part of a team which employed a thermal camera as a computer peripheral in order to
receive physiological information like blood flow, cardiac pulse and breath rate. They
monitored several participants using contact-free blood flow measurement as the basis
of a stress monitoring method. The aim was to detect elevated stress levels in HCI since
this emotion is often provoked by computer usage itself. They found out that during
mental stress there is an increased blood flow centered on the forehead above the eyes.
So they defined this area as the region of interest and put all their focus on it. They
conducted an experiment which is called the "Stroop Color Word Conflict Test" where
participants had to give the correct answer to the color displayed on the monitor which
features some difficulties for example by displaying a green color entitled as "red" or
some different title but the displayed one all monitored by the thermal camera. Adding
a time constraint which decreases as the experiment proceeds lead to elevating stress
levels and more errors so this works perfectly well to detect mental stress where an
increase in blood flow is centered on the frontal vasculature of the forehead at and just
above the corrugator or "frowning muscle" [Pav+07].
3.1.6. “Physiology-based face recognition in the thermal infrared
spectrum” by Ioannis Pavlidis et al.
The last work of Pavlidis called "Physiology-based face recognition in the thermal infrared
spectrum" features an experiment where a participant performed a running exercise as a
kind of a physical stressor. At the beginning the participant is in normal condition while
observing a change of skin temperature as the running persists. They concluded only a
negligible increase of blood volume in the periorbital region as pictured in Figure 3.5
contrary to the findings when applying an emotional stressor [Bud+07].
34
3.1. Measuring stress in the prefrontal and periorbital facial area
Figure 3.5.: The temperature in the periorbital region of the subject does not seem to
increase appreciably after physical exercise [Bud+07]
3.1.7. “Exploring the Use of Thermal Infrared Imaging in Human Stress
Research” by Veronika Engert et al.
For evaluating the thermal infrared imaging technique the paper “Exploring the Use
of Thermal Infrared Imaging in Human Stress Research” is scrutinized. It includes
the contactless way of diagnosing stress even if the participant refuses to agree to the
standard instruments of data collection and rejects the psychophysiological covariance
research. This method is noninvasive and correlates to the mood changes of the partic-
ipant but does not make it clear if anticipation, stress or a recovery phase is present.
Therefore so called stress markers are used. Heart rate, hear rate variability, finger
temperature, alpha-amylase and cortisol are perceived as stress markers. These are used
to characterize physiological states during different phases of a stress cycle.
The experiment
A pain induction method called Cold Pressor Test (CPT) where one has to place his
non-dominant hand or forearm into a tank of cold water as long as they withstand
the pain and a psychosocial challenge called Trier Social Stress Test (TSST) where
participants have a 5 minute anticipation phase followed by a stress phase in which
they are asked to give a 5 minute free speach for a simulated job interview ending the
test with a 5 minute difficult mental arithmetic have been constructed to test a wider
spectrum of stressors specifically psychosocial and physical.
An experiment was enforced with fifteen male participants where at first facial thermal
imprints in anticipation, stress and recovery phases were collected. To compare the
findings the stress markers were evaluated coming to the conclusion that the stress
markers could disambiguate anticipation, stress and recovery phases of both tests. Also
the facial thermal imprints were change-sensitive in both tests and correlated with
35
3. Related Work
Figure 3.6.: The region of interest is the supraorbital region seen in the pink colored
region inside the rectangle [SWP08]
stress-induced mood changes.
The participants underwent CPT and TSST on two consecutive days in pseudo-
randomized order leading to the result that no significant correlations between thermal
imprints and established markers were found in anticipation, stress and recovery phases.
In both tests the stress markers formed unique response profiles for all phases that
could be used to correctly predict which phase of the stress cycle a person was currently
exposed to.
Results
The overall result was the comparison of how facial thermal imprints and stress markers
performed in both tests and most thermal imprints changed significantly in both tests.
The most significant results were found in the nose tip and periorbital area but also in
the corrugator and chin areas but less in the forehead and periorbital areas. Also there
were no significant associations between change-sensitive thermal imprints and stress
markers infering that both capture unique aspects of physical and psychological stress
responses [Eng+14].
3.1.8. “Contact-free Stress Monitoring for User’s Divided Attention” by
Ioannis Pavlidis et al.
The detection of the facial region which gives the best chance for measuring stress and
the relation to attention is the most important thing in this paper work. A good hint gives
the paper "Contact-free Stress Monitoring for User’s Divided Attention" from Pavlidis
36
3.1. Measuring stress in the prefrontal and periorbital facial area
et al. It gives a detailled overview of the correlation between stress and attention,
especially when confronting one with several tasks simultaneously. It is likely that by
reaching beyond one’s certain acceptable levels this can ultimately transform into stress.
It is because of the subject’s divided attention that could lead to a degradation of one’s
performance for one or more simultaneous tasks. The supraorbital skin temperature
was used as the physiological variable of interest because of the higher measurement
sensitivity and its contact-free nature.
To simulate this situation they used simulated driving and cell phone conversation while
driving as their experimental design.
All eleven participants showed a considerable skin temperature increase in the supraor-
bital region demonstrating that concurrent performance of two critical tasks increases
user’s stress level. Further it is shown that if paying attention to more tasks it increases
stress and the temperature in the supraorbital region being the facial region of interest
when using thermal imaging for detecting stress levels as shown in Figure 3.6 [SWP08].
37
3. Related Work
3.2. Masuring stress using nasal skin temperature
In the following it is discussed if the nasal skin temperature can be used for stress
detection because the autonomic nervous system that dominates blood flow in the skin
surface part of the capillaries of the nose is activated by stressful activities such as mental
arithmetic and then the blood vessels shrink which causes a reduction of the blood flow
and then the nasal skin temperature drops.
3.2.1. “A Lifelog System for Detecting Psychological Stress with
Glass-equipped Temperature Sensors” by Hiroki Yasufuku et al.
The goal in this work has been the development of a lifelog system which features glass-
equipped sensors that can be used on a daily basis and to detect stress by examining
nasal skin temperature which is decreased by sudden stressors.
The experiment
They focused on experiments where they could test motion, environmental temperature
and stress because they lead to slopes in the nasal skin temperature. There were four
male participants with the median age of 22 years and the experiment was done for
three times per participant. One problem was that the decrease of nasal skin temperature
could not only be caused by stress but also by motion and reduction of environmental
temperature. This factors have been taken into account in designing and conducting the
experiments.
All experiments featured a resting time of ten minutes by showing a mountain stream to
stabilize the mental condition of the participants.
The first experiment had the goal of detecting stress by transcribing sentences while
listening to a distracting noise for five minutes followed by a resting period of ten
minutes.
Results
The system recognized scene of stress at a high recall rate showing a stress detection
example in Figure 3.7 [YTT16].
38
3.2. Masuring stress using nasal skin temperature
Figure 3.7.: The gray portion in the graph represents the stress detection time using the
nasal area temperature drop [YTT16]
3.2.2. “Development of a skin temperature measuring system for
non-contact stress evaluation” by Kataoka et al.
They aimed for the detection of stress from skin temperature using equipment which
continuously measures skin temperature of an object working in front of a computer
terminal.
The experiment
Therefore an experiment was conducted to investigate the relationship between stressful
tasks and skin temperature. The subjects had to follow a target on the computerscreen
with the help of a trackball and after 10 minutes there was a resting period of 5 minutes.
Additional stress was generated by using an alarm display which could only be closed by
typing in the password.
Results
The result of this is pictured in Figure 3.8 showing the nasal area temperature change
during the experiment seeing that the nasal area temperature is higher during the resting
periods and lower when doing the tasks with the additional stress situation labelled as
"emergent condition" [Kat+98].
39
3. Related Work
Figure 3.8.: Nasal area temperature change during the experiment when applying the
additional stress situation labelled as "emergent condition" [Kat+98]
40
4. Algorithm
In the following the chosen programming language and libraries are named and there
are also coding parts pictured for demonstration.
4.1. Hardware
At first the hardware is described which is essential for data collection and processing
and since there are several alternatives it is described how the selection is justified.
The optris PI160 thermal camera from the optris GmbH has been chosen since they have
experience in developing and manufacturing innovative infrared measurement devices
for non contact temperature measurement including infrared cameras of more than 10
years. Furthermore they have infrared measurement devices for different industrial
applications as well as research and development. The advantages of their technologies
is their support which comprises their free thermal analysis software, the constant
monitoring and control of virtually every manufacturing process, and reductions in
production costs through specific process optimization [the16c].
They describe their cameras as the most portable ones in the world with showing in
Figure 4.1 its measurement field [Gmb].
The characteristics of this camera are [the16a]:
• very good thermal sensitivity from 80 mK
• thermal image in real time with up to 120 Hz
• Thermal analysis kit including 3 lenses (optional)
• Detector with 160 x 120 pixels
• Compact design (dimensions: 45 x 45 x 62 mm)
• Includes license-free analysis software and full SDK
The used computer is a Dell XPS 13 Developer Edition with Intel Core i5-6200U
CPU@2.30GHz, 8GB RAM and Intel HD Graphics 520 which should offer enough
processing power for this work.
41
4. Algorithm
Figure 4.1.: Measurement field of the infrared camera optris PI [Gmb]
4.2. Implementation in C#
There were some example applications on the optris PI160 software CD which were
programmed with C# which show the thermal camera output and especially one of
them was considered as a good basis for functionality enhancement. An Image of the
basis implementation provided by Optris GmBh is to be seen in Figure 4.2.
4.3. Communication to the process imager device
Optris supplies a dynamic link library called the ImagerIPC.dll that serves the interprocess
communication (IPC) since this is handled by the Optris PI Connect software (Imager.exe)
only. In this work the DLL has been dynamically linked into the application using callback
functions for the usage of several functions of the ImagerIPC.dll which are responsible
for initiating the communication, retrieving data and setting some control parameters
[the16b].
42
4.4. Data representation of the thermal image
Figure 4.2.: The basis implementation which gets and displays the temperature
Figure 4.3.: Method for getting the temperature of a pixel
4.4. Data representation of the thermal image
The thermal camera defines the representation of the thermal images and information.
There are three ways for the data export:
43
4. Algorithm
• Colors
• Temperatures
• ADUs
The setting "Temperatures" has been chosen where the imported data can be converted
into a 160x120 pixel grayscale image via the getBitmap() method. In order to get the
temperature data the getPixelTemp() method has to be used pictured in Figure 4.3.
The array called "Values" stores temperature values of all rows of an 160x120 image.
The index is calculated like this:
• int index = (y * 160) + x;
This calculation of the index depends on the temperature mode which then computes
the temperature. There are two temperature modes which depend on the decimal places
and the one with 1 decimal place is:
• T[C] = (value - 1000)/10 with an example of:
value=1235->T=23,5C
and the one with 2 decimal places:
• T[C] = value/100 with an example of:
value=2357->T=23,57C
[the16b]
44
4.5. Implementation
Figure 4.4.: The Imager.exe software for getting the thermal image data
Figure 4.5.: CascadeClassifier holding the trained data from the XML file
4.5. Implementation
The connection between the thermal camera and the computer has to be assured with
having the Imager.exe software always running in the background which is to be seen in
Figure 4.4.
4.5.1. Facedetection with EmguCV
At first the viola-jones classifier/detector used in EmguCV for detecting faces in an image
is declared as a global object of the class CascadeClassifier. This classifier uses data
stored in an XML file to decide how to classify each image location and therefore it needs
some XML file to load trained data from. The CascadeClassifier object holds the data
loaded from this XML file as to be seen in Figure 4.5 which is done when the application
45
4. Algorithm
Figure 4.6.: Face detection algorithm "DetectMultiScale" with the parameters image,
scaleFactor, minNeighbors and optionally minSize and maxSize
is executed.
A picture of the facedetection algorithm is to be seen in Figure 4.6 where at first the
bitmap which holds the captured image information of the optris camera is converted
into an image for facilitating the face detection implementation.
The image is then converted to grey-scale so the detector can work on it.
Now DetectMultiScale is called which works on the gray-scale image. This method’s
second parameter specifies how quickly EmguCV should increase the scale for face
detections with each pass it makes over an image. Setting this higher makes the detector
run faster (by running fewer passes), but if it’s too high, it may jump too fast between
scales and miss faces whereby the default is 1.1 which means the scale increases by a
factor of 1.1 (10%) each pass. The third parameter is the minimum number of nearest
neighbors and the higher the number the fewer false positives are generated [Shi13]
[Ope08].
Then for all detected faces a rectangle is drawn and its coordinates are stored and based
on that and fixed formulas for the detection of forehead and nasal area regions the
temperatures from this areas can be processed by using the getPixelTemp method using
the x and y coordinates from the corresponding point as parameter passing.
The temperatures are then displayed and the corresponding forehead and nasal area
rectangles are drawn. This is repeated for all detected faces and after that the image is
converted back to a bitmap which is displayed in the view.
46
4.5. Implementation
Figure 4.7.: The upper part covers the temperature and live view display and the bottom
has the chart for visualization of the temperatures and the lists which hold
the experimental setup and its generated information
4.5.2. Application and its functionality
Figure 4.7 shows the final application which is divided into 2 major areas. The above
one has all the relevant information for temperature assessment and display. The target
temperature is displayed covering the whole face which is predefined in the Imager.exe
software where measured areas can be defined using x and y coordinates and the nasal
area and forehead temperatures are extracted from the red rectangles in the green box.
When starting the application the live view and the temperature processing already run.
Before starting the application in the way that the information is stored a participant
ID has to be entered in the textbox followed by hitting the "Add participant number"
button.
Then the "New participant" button is pressed which generates a randomized order of
the conditions and tasks in the left listview for the experiment. Having the order of the
experiment execution ready the "Start Data processing" starts the data collection of the
task starting the timer countdown from 3 minutes.
"Stop Data Processing" is actuated when the timer counts down to zero signaling the
push of the "Show" button which displays the recorded information in the right list view.
The "save Task" button saves this information as a text file in a desired location and the
"Start Data Processing" button can be pressed again for the execution of the next task.
The chart on the left dynamically adds the current temperatures whereby the y coordinate
47
4. Algorithm
of the chart represents the temperature scale and the x coordinate represents the time
in seconds with drawing the current temperatures every 2 seconds. Also a specific stress
significant spot or even the whole 3 minutes ongoing process can be saved as an image
by hitting the "Save Chart" button.
If the planned tasks are finished "Show for CSV" can be pressed which shows the most
vital information in the left listview and then "save Participant" saves this data as a
csv file. The "Refresh List" button helps to reset the already saved information during
a task if the participant wishes to stop or repeat it after having pressed "Stop Data
Processing".
4.5.3. Facedetection problems
In the process one major problem emerged which effects the face detection since there
are trained XML files used which work appropriately only if there are eyes in the image.
The Imager.exe software displayed the live view in green and in less detail which is a
fixed setting in the external communication tab which contains the interprocess commu-
nication(IPC) setting having selected the temperatures mode. This mode is essential for
getting the temperature.
The problem has been fixed by using glasses while recording since the glasses have a
low emissivity and are therefore displayed in black. One problem still remained with
it since the default settings in the configuration were not benefitting at all with still
noticing the face detection algorithm having problems with it. Changing the bolometer
chip temperature, emissivity, transmissivity and ambient temperature settings to high
values solved this problem. However the processed temperatures had then a slightly
higher value but since stress detection is understood as a rise in forehead temperature
and as a slope in nasal temperature this proposal has been approved.
The efficiency of the face detection algorithm also depends on the face position since
looking too much down or away results to no face detection signalising this by storing
then the temperature value "0" into the forehead and nasal area lists and also flagging
this event when exporting the result after each task as a text file. Since the temperatures
are catched and stored every 2 seconds and therefore being displayed every 2 seconds
in the chart there are vertical lines visible which indicate the stored temperature value
"0" leading to a straight downfall to value "0".
Other than that the face detection algorithm worked with high precision having a
permanent face detection rectangle in the live view which was the desired result.
48
5. User Study
There are several ways of constructing an experiment which can trigger stress. In the
following the region of interest, the experiments which were considered as ideal, the
protocol and the results are presented and discussed.
5.1. Region of interest
At first the region of interest has to be specified. And since stress detection has been most
effective in the forehead, more specifically observing the corrugator supercilii muscle
and the nasal region while doing the experiments, these two areas have been selected to
guarantuee a correct stress detection.
5.2. Experiments
The experiment is divided into three conditions and each condition itself is divided into
four tasks. Since stress isn’t triggered that easily an overall tally of twelve tasks with
each of them lasting three minutes have been considered as a high enough workload for
each participant.
I chose the conditions "Videos", "Reading Text" and "Stroop color game" where all
conditions and tasks are generated in a random order for each participant during the ex-
periment because these represent the independent variables where the order of conduct
could have some effect on the results.
After the conduct of each task from the conditions "Videos" and "Reading text" the
participants had to fill out a "PANAS" questionnaire. For each task from the condition
"Stroop color game" the participants had to fill out a "NASA Task Load Index(TLX)"
questionnaire which both enquire the participants state during the task for confirmation
of the processed results.
The first one that came in mind has been watching videos since there are several cate-
gories which trigger different human response. Therefore I opted for the video categories
49
5. User Study
"scary", "info", "boring", "entertaining" and chose for each of them one representative.
I chose the following videos, available on youtube:
• "Tom and Jerry, 64 Episode - The Duck Doctor (1952)" which I considered to be a
"entertaining" video because of many funny scenes, with having every action in
this clip being accompanied with music
• "Burning Fireplace with Crackling Fire Sounds (Full HD)" which I considered to be
a "boring" video because of only seeing the fireplace and only hearing the crackle
of the fire therefore having only focusing on little things the whole time
• "Shanghai Tower (650 meters)" which I considered to be a "scary" video starting at
minute 2:19 because of seeing two men climbing up a monstreus crane from one
of the two men’s point of view and having periods when the view is directed deep
down seeing how far they already climbed transmitting the feeling of fear, letting
people face their fear of heights, if existing
• "Was ist der Unterschied zwischen Leben und Tod?" which I considered to be a
"info" video because of hearing a clear loud voice explaining the difference between
life and death elucidating this with fitting animatic scenes facilitating the learning
process
The second condition is "Reading text" with several different categories including a
"comic", an "easy blog(article)", a "literary piece", a "science article" and chose for each
of them one representative.
I chose the following texts picking ten pages from the following sources except the
"comic" one, which will be opened via the Calibre Viewer because of the optimal quality,
and merged them into one pdf document for reading on the screen:
• "Walt Disney’s Micky Mouse", first magazine from 1. January 1966 which I consid-
ered to be a "comic" choosing the first pages
• "Psychologie Heute" which I considered to be a representative for "easy blog/article"
where I picked the article "Terror Angst"
• "Das Urteil: Eine Geschichte von Franz Kafka" which I considered to be a "literary
piece" choosing the first ten pages
• the master thesis "Simulationen - Begriffsgeschichte, Abgrenzung und Darstellung
in der wissenschafts- und technikhistorischen Forschungsliteratur" from Christiane
Spath which I considered to be a good representative for "science article" choosing
the first ten pages from the chapter "Simulationen in der wissenschafts- und
technikhistorischen Literatur"
50
5.2. Experiments
The third and last one is "Stroop color game" which tests the perceived cognitive load
with a task which can be performed in four different levels of difficulty.
I chose the android videogame "Magic Colors" which is based on the "Stroop Color Word
Conflict Test" where participants have to give the correct answer to the color displayed
on the monitor which features some difficulties for example by displaying a green color
entitled as "red" or some different title but the displayed one. "Magic Colors" features
four different difficulty levels where increasing levels lead to shorter response times.
5.2.1. Protocol
The experiment starts with recruiting twelve participants who at first give their consent
by writing for taking part in the experiment.
The participants sit in front of a table upright on a chair with their hands on the table.
The arrangement of the devices has two key distances with the first one being the height
in which the thermal camera is deployed which should be located on the table directed
to the participant’s face to ensure that they are in the camera’s exposing field. However
the height of the camera could be adjusted depending on the size of the participant. The
second important distance is the one between the thermal camera and the head of the
participants which optimally is at least one metre.
The setup starts with a short test if all the devices work. After that the study starts with
randomizing all conditions and tasks for each participant which defines the order in
which the tasks are performed. I asked the subjects to concentrate during the tasks
and after each task they filled out a questionnaire to catch their feelings and state they
endured while doing the task.
The experiment is concluded with an interview, a debriefing and a compensation of 10
euros where the receipt of the money is confirmed by writing.
5.2.2. Questionnaires
The application processes the incoming temperatures during the experiments and decides
on the basis of the related work if stress emerged. For result confirmation in the
conditions "Videos" and "Reading Text" I chose a "PANAS" questionnaire which enquires
the extent the participant felt over the past task with state of mind terms like "interested",
"distressed" or "upset" and possible selection between five states starting at "very slightly
or not at all" and finishing at "extremely".
For the "Stroop color game" result confirmation I chose the "NASA TLX" questionnaire
which enquires to click on each scale at the point that best indicates the participants
experience of the task by having terms like "Mental Demand" or "Frustration" and the
possibility of choosing rank on to ten beginning at "small" and finishing at "high".
51
5. User Study
5.3. Results
The desired results should show a clear and definite outcome, therefore I considered
to use descriptive and inferential statistics in Microsoft Excel 2016 having all occuring
terms listed in the "Glossar" [RS16].
Descriptive statistics
Descriptive statistics contain measures, like calculated means, data range and standard
deviations making it relatively easy to visualize the experiment’s results. Especially the
mean and the sum are scrutinized explicitly since in the calculation of these two all
values have been considered and therefore they show the partially significant differences
optimally as decimals. Also when there is a high mean and sum, that values affect the
other values greatly. So in general having a high mean and sum indicate that the bulk
of all values is a high temperature difference value and therefore one can infer that a
high stress and attention level existed. In the following I will show the outcome of each
condition separately [Met16a].
Inferential statistics
As a representative for inferential statistics we chose the one-way analysis of vari-
ance(ANOVA). With this analysis it is possible to compare means of three or more
samples examining if the condition’s order and task order within each condition strongly
differ from one another.
There is the null hypothesis which would be that all conditions or tasks within their con-
dition produce the same response, on average, meaning that no significant differences
exist between them regarding their order which was randomized.
The critical value, in this case F critical, is important since it is the number that the test
statistic, in this case the P-value and F, must exceed to reject the test [Met16b] [sta16].
52
5.3. Results
Figure 5.1.: Tasks "scary" and "info" indicate the lowest mean and sum for this condition
Figure 5.2.: Tasks "boring" and "entertaining" indicate the highest measured stress and
attention levels for this condition
Condition "Videos"
As to be seen in Figure 5.1 and Figure 5.2 especially the tasks "boring" and "entertaining"
can be highlighted having the highest mean and sum values indicating the greatest
measured stress and attention levels there.
53
5. User Study
Figure 5.3.: Task "comic" having high values in every department
Figure 5.4.: Tasks "literary piece" and "science article" strongly differ whereby task
"science article" showing the most clear results
Condition "Reading Text"
Figure 5.3 and Figure 5.4 show that the tasks "comic" and "easy blog" do not differ
that much, generating almost equally high values but task "science article" has to be
highlighted as the most efficient task as to be seen in its high mean and sum.
54
5.3. Results
Figure 5.5.: Tasks "level 1" and "level 2" having high values throughout
Figure 5.6.: Highlighting task "level 3" with the greatest results of the entire experiment
Condition "Stroop Color game"
Figure 5.5 and Figure 5.6 feature very high values in every section of each task empha-
sizing tasks "level 1" and "level 2" and especially "level 3" with the highest mean and
sum of the entire experiment.
55
5. User Study
Figure 5.7.: The order of the conditions generated with a random algorithm observ-
ing that the "Stroop color game" condition was executed mostly in the
beginning, followed by "Videos" and "Reading Text"
Figure 5.8.: Checking if all conditions are randomized equally being called the null
hypothesis, observing that the P-value and F do not exceed F critical so H0
is accepted and not rejected
Analyzing condition order
The null hypothesis, denoted H0, for the overall F-test for this experiment would be
that all three conditions produce the same response, on average. The summary of
the calculation is to be seen in Figure 5.7 where the count is defined by the number
of participants, a small sum reveals that the condition was performed mostly at the
beginning and a higher sum shows that the condition was performed mostly at the end
of the experiment.
Figure 5.8 shows the ANOVA results for all conditions where it is observable that the
P-value and F do not exceed F critical so H0 is accepted and not rejected.
56
5.3. Results
Figure 5.9.: Checking if all tasks within their condition have been randomized equally
observing that the tasks "boring", "entertaining", "comic", "easy blog", "level
3" and "level 4" were mostly done at the end of their respective condition
Figure 5.10.: Checking if all tasks within their conditions are randomized equally being
called the null hypothesis, observing that the P-value and F do not exceed
F critical so H0 is accepted and not rejected
Analyzing the order of tasks within each condition
The null hypothesis, denoted H0, for the overall F-test for this experiment would be that
all four tasks within the three conditions produce the same response, on average. The
summary of the calculation is to be seen in Figure 5.9 where the count is defined by the
number of participants, a small sum reveals that the condition was performed mostly
at the beginning and a higher sum shows that the condition was performed mostly at
the end of the experiment. Figure 5.10 shows the ANOVA results for tasks within each
condition where it is observable that the P-value and F do not exceed F critical so H0 is
accepted and not rejected.
57
5. User Study
Figure 5.11.: An order of the most stress and attention generating tasks observing that
the "Stroop color game" condition’s tasks had the best results, followed by
"Videos" and "Reading Text"
5.3.1. Discussion
Statistics helped to analyze the processed data leading to the task order in Figure 5.11
where a high mean und sum lead up to high stress and attention, since this is detected
when the temperature gap between forehead and nasal area temperatures is getting
bigger, so adding this big temperature gap values up results into a high sum and mean.
This section will help to understand these results with participant’s statements and the
evaluation of the questionnaires while I will also show some interesting temperature
flows illustrated via charts.
The charts x-coordinate and y-coordinate feature the time in seconds and temperature
in celsius respectively. The blue and red lines stand for the forhead and nasal area
temperatures respectively. This is the best way to visualize high stress and attention
levels since this pertains when the forehead temperature rises and the nasal area
temperature decreases observing a bigger temperature gap between these two.
The procedure in finding the right explanations to the outcomes in Figure 5.11 contains
three steps of examination:
• the condition’s order in Figure 5.7
• the task’s order within their condition in Figure 5.9
• participant’s feedback and filled out questionnaires
58
5.3. Results
These three steps are necessary since the condition’s order and the order of the tasks
within their condition represent the independent variables in this experiment where the
order of conduct could have had an impact on this result and analyzing the confirmation
from the subjects directly is a optimal confirmation source to understand the results.
Condition "Stroop color game"
In general all participants except one reported big frustration and being highly attentive
to the "Stroop color game" because of being asked by myself to try giving their best.
Figure 5.11 shows that task "level 3" had the best stress detection results of the entire
experiment so this will be explained.
1. Figure 5.7 shows that the "Stroop color game" condition has been performed mostly
at the beginning of the experiment.
So this condition has been performed mostly when the participants have been probably
in a relaxed state.
2. Figure 5.9 shows that the tasks "level 3" and "level 4" have been performed mostly at
the end of their condition.
So the tasks have been mostly done when the participants have been already under huge
stress and in a very attentive state because of he previous played levels. But because
Figure 5.11 shows that "level 4" was the least effective task of the "Stroop color game"
condition generating the least stress and attention the conjecture, that the order of
conduct could explain the great results of "level 3", has to be defused.
3. Based on the participant’s questionnaires and assertions for "level 3" they were really
trying to set a high score and favoured the response time of that level the most because
of being not too fast nor too slow and therefore having most fun playing that level.
Many reported that "level 4" was too tough not leading to frustration but disinterest and
therefore less attention and stress.
To give an example of the "Stroop color game" condition’s power to generate high
attention and stress the results of participant 3 who had the order of "level 3", "level 2",
"level 1" and "level 4" are shown in Figure 5.12, Figure 5.13, Figure 5.14 and Figure
5.15.
59
5. User Study
Figure 5.12.: Participant 3: Started in a relaxed state playing level 3 observing emerging
stress as the game prolonged
Figure 5.13.: Participant 3: Remaining in this stressful state while playing level 2 seeing
the temperature gap getting bigger
60
5.3. Results
Figure 5.14.: Participant 3: Seeing the participant getting irritated because of the
performance loss while playing level 1, only being able to get low scores
on the easiest difficulty
Figure 5.15.: Participant 3: Achieving even bigger stress while playing level 4 seeing the
participant being stressed out observing his lowest nasal area temperatures
in this experiment
61
5. User Study
Condition "Reading Text"
Another big result has been monitored in the "science article" condition.
1. Figure 5.7 shows that the "Reading Text" condition has been performed mostly at the
end of the experiment.
So this condition has been performed mostly when the participants may have been
probably in an exhausted or stressful state.
Figure 5.11 shows that the other conditions generated by far more stress than the
"Reading Text" condition so the participants may have been indeed in an exhausted state
when doing the "Reading Text" condition.
But since the "science article" task performed so well this has to be analysed in more
detail.
2. Figure 5.9 shows that the "science article" task was done more in the beginning or in
the middle of its condition and the "literary piece", which had the worst results according
to Figure 5.11, was done mostly prior to the "science article" task indicating that the
participants entered this task in a relaxed state, because of having "literary piece" prior
to it which did not stress the participants at all.
3. The answer lies again in the participant’s comments with some saying that the text’s
complexity really forced them to enter a high attentive state, having to think big about
its sense and the few english passages exacerated to remain in a relaxed state.
Condition "Videos"
The "Videos" condition contains the second best results, especially the tasks "boring" and
"entertaining".
1. Figure 5.7 shows that this condition has been performed mostly in the middle or at
the end of the experiment mostly after the "Stroop color game" condition.
It is possible that playing the "Stroop color game" just prior to the "Videos" condition
resulted in entering the "Videos" condition in an already stressful state, therefore affecting
the outcome.
2. Figure 5.9 shows that the tasks "boring" and "entertaining" have been both done
mostly at the end of their "Reading Text" condition since having the same sum and
average values.
So it could be argued that the previous tasks, especially the previous videos may had an
effect on the "boring" and "entertaining" tasks, too. The next task which comes close to
their sum and average values is "scary" having only a slightly lower sum and average
indicating that "scary" has been performed mostly prior to the "boring" and "entertaining"
tasks.
3. Many participants reported feeling stressed while watching the "scary" video because
of the good video quality and ego perspective view, making it possible to think in the
62
5.3. Results
climber’s shoes and since Figure 5.11 shows that the "scary" task differs only little from
the "boring" and "entertaining" videos therefore this assumption can not be suspended.
Again the participant’s feedback makes the result plain since they reported of being
not that attentive to the "boring" video leading to rambling thoughts, switching their
attention to things that were bothering them.
The "entertaining" video’s feedback was quite the opposite because of most participants
reporting high attention levels, liking the video confirming the processed data.
63

6. Summary and outlook
This chapter provides a summary of the aims of this work followed by a conclusion and
future outlook for possible changes and improvements of this work.
6.1. Summary
In this thesis I explored the relation between stress and attention by using thermal imag-
ing technology to introduce a non-invasive, efficient, real-time and robust interaction
technique. Moreover I illustrated which actions or rather tasks help to generate artificial
stress and thereby increase attention.
By using such a broad spectrum of common stress generating tasks from different areas
a comprehensive observation has been possible. The fact that attention promoting tasks
provoke stress has been used when creating in the setup making use of some ideas from
other sources but also inventing a totally new setup which is ideal for the scrutiny of
the stress attention relation. By using well-known computer vision techniques I built a
camera-projector system that can monitor users in front of the thermal camera avoiding
the inclusion of traditional RGB or depth cameras by processing all information using
only the thermal image. Also some problems like for instance the illumination depen-
dency is being avoided by using thermal cameras, therefore encouraging the robustness.
This new interaction system provides the following advantages:
• The thermal camera’s robustness guarantees an usage in any lightning condition,
even in dark environment
• The illumination independency leads to a drastically reduced complexity of face
detection
• It can be used in daily life because of it’s portability und utility
• Efficiency of processing stress related data in a non-invasive way
65
6. Summary and outlook
The first goal has been the detection of the areas on a human body which fit the bill
for stress detection using a thermal imaging camera. Since the face is visible and the
thermal camera can only measure areas which are uncovered the face seemed as the
best solution. Another aim was detecting stress in a non-invasive way abandoning the
usage of wires or other things that could disturb the participants. It was wished that the
camera is arranged in front of the participant by a long enough way so the participant
remains in a relaxed state concentrating only on the experiment. In order to achieve the
goals big research had to be done for spotting the right facial areas for stress detection.
By using the nasal area and the forehead area, particularly the corrugator supercilii
muscle right above the eye, this aim was achieved with having to implement this in the
most successful and flawless way. After achieving this goals the main purpose of this
work, namely the correlation between attention and stress could be investigated in a
comprehensive user study involving 12 participants. Furthermore the user study showed
the relation between stress and attention citing many stress and attention enhancing
tasks observing that attention is a necessity for stress advancement since this includes the
concentration on a subject and the other way stress was monitored when high attention
levels have been reported.
6.2. Conclusion
The application which should display a thermal image in an only possible indirect way
by using the inter process communication(IPC) with the optris Imager.exe has been
developed. It was possible to detect faces in a high efficient way but only when having
the eyes visible in the thermal image since a thermal image does not have the quality
of a normal image making it harder for the algorithm to detect a face. The user study
showed that the face detection algorithm worked on some participants not wearing
glasses at first but even then in an inaccurate way seeing the forehead and nasal area
rectangles slightly displaced which would lead to false results. By using glasses this
problem was fixed and so this was not considered as a problem by itself.
The usage of the optris Imager.exe brought a problem in the configuration settings be-
cause the device settings influenced the thermal image quality and only when switching
the values to high ones the quality was good enough for the face detection algorithm to
work on. But since only the rise of the forehead temperatures and the drop of nasal area
temperatures indicate incoming stress this was approved.
The experiment encompassed 12 participants from different ages who performed overall
12 tasks ranging from watching videos, reading texts and playing a game. Especially
when playing the game high attention and stress levels have been processed and reported
seeing this tasks as a success and helping to solve this topic. But also when watching
videos and reading texts some interesting outcomes were noticed which were possible
66
6.3. Potential field of application
due to having little technological problems with the thermal camera and the application
when doing the experiment.
Synoptic consistent results during the experiment were monitored leading to a guarantee
that the findings did not result from a coincidence making it possible to verify it.
6.3. Potential field of application
The developed interaction system measures emerging stress by processing the forehead
and nasal area temperatures by calculating the difference between the forehead and
nasal area temperatures. This section discusses potential application ideas in which
stress detection can enlarge the interaction space inspired by the user study results,
too.
Effect of motion picture
The experiment featured the "Videos" condition which triggered high attention and
stress levels based on the topic. Since this approach is non-invasive and the subjects
can therefore concentrate fully on the video this application could be used as a second
confirmation source alongside their feedback in a preview. The desired impact of some
scenes can be observed, the length of scenes can be tested by looking at the amount of
attention in the course of watching them.
Effect of video games
The experiment featured the "Stroop color game" condition which triggered high atten-
tion and stress levels based on the level of difficulty. One big result in the user study
has been that the third level of the game featured the biggest stress and attention levels
indicating that its response time was perfectly defined. Based on that the application
could be used in an early video game testing phase seeing if the desired effect, like low
frustration levels when playing the first levels, turns up. By emerging stress the artificial
intelligence of the opponents in the game could be adjusted automatically.
Effect of graphical user interfaces
The complexity of graphical user interfaces could be tested and since everybody has
his own taste the arrangement could be automatically adjusted by displaying less
67
6. Summary and outlook
unimportant elements but therefore increasing the nesting or displaying help texts and
mouse-hover messages.
Effect of learning applications
There are many learning softwares which promise the best learning success but since
the user is in the limelight the software should therefore react way more to changes of
state where the amount of stress and attention the user pays to the software indicates a
recurrent usage.
The measured stress could decide the level of cognitive workload and based on that the
level of difficulty of the next task could be calculated.
Effect of texts
The experiment showed great results in the "science article" task which generated big
stress and high attention to it because of its complexity in structure and content but also
because of being written in two languages.
So a programm which displays texts could display additional information, explanations
or definitions when registering stress making it easier for the user to read it.
Outlook
At first the accuracy of the application could be improved with for example storing the
temperatures every second or even in smaller intervals, excluding the rounding of the
stored temperatures.
The performance could be improved by using a even better thermal camera or by using
a better thermal camera software if available. More efficient face detection algorithms
could augment the application and make it unnecessary to wear glasses with this
depending on the participant and the therefore given facial quality in a thermal image
resulting into sometimes good and sometimes bad face detection without using glasses.
It was possible to find out a correlation between attention and stress seeing this work as
a success admitting at the same time that the experiment could get even better results
by using future and even today’s available technology, especially virtual reality which is
seen as the most important new technology since the invention of the internet, enabling
almost real life experiences which could lead to the most accurate findings when using
this application.
68
A. Glossar
A.1. Definitions
Far Infrared
is a region in the infrared spectrum of electromagnetic radiation
Human Computer Interaction
researches the design and use of computer technology focusing on the interfaces
between people and computers
Evaporograph technique
a method used for making far-infrared radiation visible
Homeostasis
is the property of a system in which a variable is actively regulated to remain very
nearly constant
Pathology
is a significant component of the causal study of disease and a major field in
modern medicine and diagnosis.
Electro-optics
belongs to electrical engineering and material physics and involves components,
devices and systems which operate by the propagation and interaction of light
with various tailored materials
thermal imaging
is an example of infrared imaging science
hyperspectral imaging
collects and processes information from across the electromagnetic spectrum to
obtain the spectrum for each pixel in the image of a scene with the purpose of
finding objects, identifying materials or detecting processes
69
A. Glossar
electromagnetic spectrum
includes all possible frequencies of electromagnetic radiation and also stands for
the characteristic distribution of electromagnetic radiation emitted or absorbed by
a particular object
physiological
is the scientific study of the normal function in living systems and focuses in how
organisms, organ systems, organs, cells and biomolecules carry out the chemical
or physical functions that exist in a living system
electrodermal activity
is the property of the human body that causes continuous variation in the electrical
characteristics of the skin and leads to skin conductance because of sweating when
one feels psychological or physiological aroused
haemoglobin oxygenation level
is the amount of oxygen in the red blood cells to enable observation of different
active areas of the brain or other organs
alpha-amylase
is a salivary enzyme which reacts heavily to a emotional stressor but does not corre-
late to other stress markers and therefore is very useful for detecting physiological
changes due to stress
cortisol
is a steroid hormone and is released in response to stress and low blood-glucose
concentration
acute stress
temporary state of arousal with typically clear start and endpattern
chronical stress
state of persistent excitement where the requirements outshine the inner and outer
resources for overcoming it
stressor
is a stimulusevent featuring many extern and intern conditions causing some
reaction, an adjustment of one’s behavior, physiological, emotional and cognitive
states to the caused imbalance of the organism and the ability of overcoming it
cognition
is the mental action or process of acquiring knowledge and understanding through
thought, experience and the senses
70
A.1. Definitions
hypothalamus
is a portion of the brain that controls body temperature, hunger, important aspects
of parenting and attachment behaviors, thirst, fatique, sleep and circadian rhythms
autonomous nerve system
is a division of the peripheral nervous system that influences the function of
internal organs regulating bodily functions like heart rate, digestion, respiratory
rate, pupillary response, urination, sexual arousal and is in control of the fight-or-
flight response and the freeze-and-dissociate response.
hypophysis
is a protrusion off the bottom of the hypothalamus at the base of the brain regulat-
ing stress, growth, reproduction, lactation and helping to control blood pressure,
certain functions of the sex organs, thyroid glands, metabolism, pregnancy, child-
birth, nursing, water/salt concentration, kidneys, temperature regulation and pain
relief.
pharynx
is the part of the throat behind the mouth and nasal cavity being part of the
digestive system and also the conducting zone of the respiratory system
adrenal glands
are endocrine glands located above the kidneys that produce a variety of hor-
mones including adrenaline and the steroids aldosterone and cortisol where each
gland has also an inner medulla which has a function to produe a rapid response
throughout the body in stress situations
adrenaline
is primarily a medication and a hormone which features side effects including a
fast heart rate and high blood pressure
noradrenalin
is an organic chemical that functions in the human rain and body as a hormone
and neurotransmitter having the general function of mobilizing the brain and
body for action and is released in much higher levels during situations of stress or
danger, in so-called fight-or-flight response
bone marrow
is the flexible tissue in the interior of bones
red blood cell
is responsible for blood clotting in case of an injury
white blood cell
fights infections
71
A. Glossar
thyroid
located at the front of the neck below the laryngeal prominence is an endocrine
gland in the body and consists of two connected lobes which secretes thyroid
hormones that influence the metabolic rate, protein synthesis and have a wide
range of other effects, including on development
adrenal cortex
mediates the stress response through the production of mineralocorticoids an
glucocorticoids such as aldosterone and cortisol and is located along the perimeter
of the adrenal gland
working memory
is a cognitive system with a limited capacity that is responsible for the transient
holding, processing and manipulation of information also being an important
process for reasoning and the guidance of decision making and behaviour
brainstem
is the superior part of the brain, adjoining and structurally continuous with the
spinal cord providing the main motor and sensory innervation to the face and neck
via the cranial nerves and has many basic functions including heart rate, breathing,
sleeping and eating
myelinated vagus
based on a theory and specifies two functionally distinct branches of the vagus
nerve which serve different evolutionary stress responses in mammals
autism
is a neurodevelopmental disorder characterized by impaired social interaction,
verbal and non-verbal communication and also restricted and repetitive behaviour
schizophrenia
is a mental disorder characterized by abnormal social behaviour and failure to
understand what is real
supraorbital
refers to the region immediately above the eye sockets where in humans the
eyebrows are located
periorbital
pertaining to the area surrounding the socket of the eye
Corrugator supercilii muscle
is a small, narrow, pyramidal muscle close to the eye and draws the eyebrow
downward and medially, producing the vertical wrinkles of the forehead and may
be regarded as the principle muscle in the expression of suffering
72
A.1. Definitions
procerus muscle
is a small pyramidal slip of muscle deep to the superior orbital nerve, artery and
vein
orbicularis oculi muscle
is a muscle in the face that closes the eyelids
Mean
is the average of all the numbers by adding all of the numbers and then dividing
the sum by the total count of the numbers and is sometimes called the arithmetic
mean
Standard error
is the standard deviation of the sampling distribution of a statistic, most commonly
of the mean
Median
is the middle number in a sequence of numbers ordered by size
Modus
is the number which occurs most often within a set of numbers
Standard deviation
is a measure that is used to quantify the amount of variation or dispersion of a set
of data values
Sampling variance
is the expectation of the squared deviation of a random variable from its mean and
it informally measures how far a set of (random) numbers are spread out from
their mean
Kurtosis
is a measure of the "tailedness" of the probability distribution of a real-valued
random variable
Skew
is a measure of the asymmetry of the probability distribution of a real-valued
random variable about its mean
Value range
is the difference between the highest and lowest values within a set of numbers
Sum
is the result when adding all the temperature differences (forehead temperature -
nasal area temperature)
73
A. Glossar
Count
is the number of all temperature difference values for each task that have been
processed
Average
is the sum of a list of numbers divided by the count of numbers in the list
Standard error: is the standard deviation of the sampling distribution of a statistic,
most commonly of the mean
Variance
is the expectation of the squared deviation of a random variable from its mean,
and it informally measures how far a set of (random) numbers are spread out from
their mean
Sum of Squares (SS)
The sum of squares represents a measure of variation or deviation from the mean
and is calculated as a summation of the squares of the differences from the mean.
The calculation of the total sum of squares considers both the sum of squares from
the factors and from randomness or error helping in ANOVA to express the total
variation that can be attributed to various factors
degrees of freedom (df)
is the number of values in the final calculation of a statistic that are free to vary
more precise it can be defined as the minimum number of independent coordinates
that can specify the position of the system completely
Mean squares (MS)
represents an estimate of population variance and is calculated by dividing the
corresponding sum of squares by the degrees of freedom helping in ANOVA to
determine whether factors are significant
F
frequently arises as the null distribution of a test statistic, most notably in the
analysis of variance
P-value
is a function of the observed sample results (a test statistic) relative to a statistical
model, which measures how extreme the observation is
F critical
a critical value is a point on the test distribution that is compared to the test
statistic to determine whether to reject the null hypothesis. If the absolute value
of the test statistic is greater than the critical value, one can declare statistical
significance and reject the null hypothesis
74
Bibliography
[AGH98] M Anbar, B. Gratt, D Hong. “Thermology and facial telethermography.
Part I: history and technical review.” In: Dentomaxillofacial Radiology 27.2
(1998), pp. 61–67 (cit. on p. 11).
[Bud+07] P. Buddharaju, I. T. Pavlidis, P. Tsiamyrtzis, M. Bazakos. “Physiology-based
face recognition in the thermal infrared spectrum.” In: Pattern Analysis
and Machine Intelligence, IEEE Transactions on 29.4 (2007), pp. 613–626
(cit. on pp. 34, 35).
[CG80] M. Creates, J Grayson. “The effect of adrenaline and noradrenaline on
coronary vascular reserve in the dog.” In: The Journal of physiology 309
(1980), p. 557 (cit. on p. 21).
[CJDM07] S. Cohen, D. Janicki-Deverts, G. E. Miller. “Psychological stress and dis-
ease.” In: Jama 298.14 (2007), pp. 1685–1687 (cit. on p. 12).
[CTN05] R. J. Cook, S. Thakore, N. M. Nichol. “Thermal imaging–a hotspot for the
future?: A case report.” In: Injury Extra 36.9 (2005), pp. 395–397 (cit. on
p. 11).
[DC95] A. Di Carlo. “Thermography and the possibilities for its applications in
clinical and experimental dermatology.” In: Clinics in dermatology 13.4
(1995), pp. 329–336 (cit. on p. 11).
[Ebi+12] S. J. Ebisch, T. Aureli, D. Bafunno, D. Cardone, G. L. Romani, A. Merla.
“Mother and child in synchrony: thermal facial imprints of autonomic
contagion.” In: Biological psychology 89.1 (2012), pp. 123–129 (cit. on
p. 11).
[Eng+14] V. Engert, A. Merla, J. A. Grant, D. Cardone, A. Tusche, T. Singer. “Exploring
the use of thermal infrared imaging in human stress research.” In: PloS one
9.3 (2014), e90782 (cit. on pp. 16, 36).
[GZ08] R. J. Gerrig, P. Zimbardo. “Psychologie (18. Auflage).” In: Hallbergmoos:
Pearson Deutschland GmbH (2008) (cit. on p. 17).
[Gar+07] M. Garbey, N. Sun, A. Merla, I. Pavlidis. “Contact-free measurement of car-
diac pulse based on the analysis of thermal imagery.” In: IEEE Transactions
on Biomedical Engineering 54.8 (2007), pp. 1418–1426 (cit. on p. 11).
75
Bibliography
[Gmb] O. GmbH. “Operator’s Manual optris PI.” In: optris infrared thermometers ()
(cit. on pp. 15–17, 41, 42).
[Hah+12] A. C. Hahn, R. D. Whitehead, M. Albrecht, C. E. Lefevre, D. I. Perrett. “Hot
or not? Thermal reactions to social contact.” In: Biology letters (2012),
rsbl20120338 (cit. on p. 11).
[Hon+09] K. Hong, P. Yuen, T. Chen, A. Tsitiridis, F. Kam, J. Jackman, D. James,
M. Richardson, W. Oxford, J. Piper, et al. “Detection and classification of
stress using thermal imaging technique.” In: SPIE Europe Security+ Defence.
International Society for Optics and Photonics. 2009, pp. 74860I–74860I
(cit. on pp. 21, 31–33).
[Hsi+90] J. Hsieh, K. Chan, P. Lui, T. Lee. “Clinical application of infrared thermog-
raphy in diagnosis and therapeutic assessment of vascular ischemic pain.”
In: Ma zui xue za zhi= Anaesthesiologica Sinica 28.4 (1990), pp. 493–501
(cit. on p. 11).
[Jon98] B. F. Jones. “A reappraisal of the use of infrared thermal image analysis in
medicine.” In: IEEE transactions on medical imaging 17.6 (1998), pp. 1019–
1027 (cit. on p. 11).
[KRC81] R. Katz, K. Roth, B. Carroll. “Acute and chronic stress effects on open field
activity in the rat: implications for a model of depression.” In: Neuroscience
& Biobehavioral Reviews 5.2 (1981), pp. 247–251 (cit. on p. 18).
[Kat+98] H. Kataoka, H. Kano, H. Yoshida, A. Saijo, M. Yasuda, M. Osumi. “Devel-
opment of a skin temperature measuring system for non-contact stress
evaluation.” In: Engineering in Medicine and Biology Society, 1998. Proceed-
ings of the 20th Annual International Conference of the IEEE. Vol. 2. IEEE.
1998, pp. 940–943 (cit. on pp. 39, 40).
[LB97] J. Lynch, K. M. Brown. “Ethylene and plant responses to nutritional stress.”
In: Physiologia Plantarum 100.3 (1997), pp. 613–619 (cit. on p. 18).
[LPC01] J. A. Levine, I. Pavlidis, M. Cooper. “The face of fear.” In: The Lancet
357.9270 (2001), p. 1757 (cit. on pp. 20, 29, 30).
[Lar+11] E. Larson, G. Cohn, S. Gupta, X. Ren, B. Harrison, D. Fox, S. Patel. “Heat-
Wave: thermal imaging for surface user interaction.” In: Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems. ACM. 2011,
pp. 2565–2574 (cit. on p. 12).
[Law56] R. Lawson. “Implications of surface temperatures in the diagnosis of breast
cancer.” In: Canadian Medical Association Journal 75.4 (1956), p. 309
(cit. on p. 11).
76
Bibliography
[LeB09] V. R. LeBlanc. “The effects of acute stress on performance: implications
for health professions education.” In: Academic Medicine 84.10 (2009),
S25–S33 (cit. on p. 19).
[MR07] A. Merla, G. L. Romani. “Thermal signatures of emotional arousal: a func-
tional infrared imaging study.” In: (2007), pp. 247–249 (cit. on p. 20).
[Met16a] R. Methods. http://www.socialresearchmethods.net/kb/statdesc.php. WEB
CENTER FOR SOCIAL RESEARCH METHODS, 2016 (cit. on p. 52).
[Met16b] R. Methods. http://www.socialresearchmethods.net/kb/statinf.php. WEB
CENTER FOR SOCIAL RESEARCH METHODS, 2016 (cit. on p. 52).
[Mic03] Microsoft. https://msdn.microsoft.com/en-us/library/aa664274(v=vs.71).aspx.
Microsoft Developer Network, 2003 (cit. on p. 22).
[Mic15a] Microsoft. https://msdn.microsoft.com/en-us/library/dn762121.aspx. Mi-
crosoft Developer Network, 2015 (cit. on p. 23).
[Mic15b] Microsoft. https://msdn.microsoft.com/en-us/library/z1zx9t92.aspx. Mi-
crosoft Developer Network, 2015 (cit. on p. 22).
[Mic15c] Microsoft. https://msdn.microsoft.com/en-us/library/z1zx9t92.aspx. Mi-
crosoft Developer Network, 2015 (cit. on p. 23).
[NC10a] B. R. Nhan, T. Chau. “Classifying affective states using thermal infrared
imaging of the human face.” In: IEEE Transactions on Biomedical Engineering
57.4 (2010), pp. 979–987 (cit. on p. 11).
[NC10b] B. R. Nhan, T. Chau. “Classifying affective states using thermal infrared
imaging of the human face.” In: IEEE Transactions on Biomedical Engineering
57.4 (2010), pp. 979–987 (cit. on p. 20).
[Nak+05] K. Nakayama, S. Goto, K. Kuraoka, K. Nakamura. “Decrease in nasal tem-
perature of rhesus monkeys (Macaca mulatta) in negative emotional state.”
In: Physiology & behavior 84.5 (2005), pp. 783–790 (cit. on p. 20).
[Ope08] L. OpenCV. “Computer vision with the OpenCV library.” In: GaryBradski &
Adrian Kaebler-O’Reilly (2008) (cit. on pp. 23, 46).
[PL02] I. Pavlidis, J. Levine. “Thermal image analysis for polygraph testing.” In:
Engineering in Medicine and Biology Magazine, IEEE 21.6 (2002), pp. 56–64
(cit. on p. 33).
[PLB01] I. Pavlidis, J. Levine, P. Baukol. “Thermal image analysis for anxiety detec-
tion.” In: 2 (2001), pp. 315–318 (cit. on p. 20).
[Pav+07] I. Pavlidis, J. Dowdall, N. Sun, C. Puri, J. Fei, M. Garbey. “Interacting with
human physiology.” In: Computer Vision and Image Understanding 108.1
(2007), pp. 150–170 (cit. on p. 34).
77
Bibliography
[Pav+12] I. Pavlidis, P. Tsiamyrtzis, D. Shastri, A. Wesley, Y. Zhou, P. Lindner, P. Bud-
dharaju, R. Joseph, A. Mandapati, B. Dunkin, et al. “Fast by nature-how
stress patterns define human experience and performance in dexterous
tasks.” In: Scientific Reports 2 (2012) (cit. on p. 11).
[Por01a] S. W. Porges. “The polyvagal theory: phylogenetic substrates of a social
nervous system.” In: International Journal of Psychophysiology 42.2 (2001),
pp. 123–146 (cit. on p. 11).
[Por01b] S. W. Porges. “The polyvagal theory: phylogenetic substrates of a social
nervous system.” In: International Journal of Psychophysiology 42.2 (2001),
pp. 123–146 (cit. on p. 20).
[Pur+05a] C. Puri, L. Olson, I. Pavlidis, J. Levine, J. Starren. “StressCam: non-contact
measurement of users’ emotional states through thermal imaging.” In:
CHI’05 extended abstracts on Human factors in computing systems. ACM.
2005, pp. 1725–1728 (cit. on p. 11).
[Pur+05b] C. Puri, L. Olson, I. Pavlidis, J. Levine, J. Starren. “StressCam: non-contact
measurement of users’ emotional states through thermal imaging.” In:
(2005), pp. 1725–1728 (cit. on p. 20).
[RA12] E. Ring, K Ammer. “Infrared thermal imaging in medicine.” In: Physiological
measurement 33.3 (2012), R33 (cit. on p. 11).
[RS16] S. L. Roger Stern Ian Dale. http://www.reading.ac.uk/ssc/resources/Docs/StatisticalGlossary.pdf .
University of Reading, 2016 (cit. on p. 52).
[Rin00] E. Ring. “The discovery of infrared radiation in 1800.” In: Imaging Science
Journal 48.1 (2000), pp. 1–8 (cit. on p. 11).
[Rin04] E. Ring. “The historical development of thermal imaging in medicine.” In:
Rheumatology 43.6 (2004), pp. 800–802 (cit. on p. 11).
[Rin90] E. Ring. “Quantitative thermal imaging.” In: Clinical Physics and Physiologi-
cal Measurement 11.4A (1990), p. 87 (cit. on p. 11).
[SWP08] D. Shastri, A. Wesley, I. Pavlidis. Contact-free Stress Monitoring for User’s
Divided Attention. INTECH Open Access Publisher, 2008 (cit. on pp. 36,
37).
[Sel13] H. Selye. Stress in health and disease. Butterworth-Heinemann, 2013 (cit.
on p. 19).
[Sel50] H. Selye. “The physiology and pathology of exposure to stress.” In: (1950)
(cit. on p. 18).
[Sel56] H. Selye. “The stress of life.” In: (1956) (cit. on p. 19).
[Sel76] H. Selye. “Stress without distress.” In: Psychopathology of human adaptation.
Springer, 1976, pp. 137–146 (cit. on p. 17).
78
Bibliography
[Shi13] S. Shi. Emgu CV Essentials. Packt Publishing Ltd, 2013 (cit. on pp. 23–27,
46).
[Ski+07] J. J. Skitzki, Q. Chen, W. Wang, S. S. Evans. “Primary immune surveillance:
some like it hot.” In: Journal of Molecular Medicine 85.12 (2007), pp. 1361–
1367 (cit. on p. 11).
[Sta04] M. A. Staal. “Stress, cognition, and human performance: A literature review
and conceptual framework.” In: (2004) (cit. on p. 19).
[Sto00] R. Stone. “Stress: The invisible hand in Eastern Europe’s death rates.” In:
Science 288.5472 (2000), pp. 1732–1733 (cit. on p. 18).
[Tay+00] S. E. Taylor, L. C. Klein, B. P. Lewis, T. L. Gruenewald, R. A. Gurung, J. A. Up-
degraff. “Biobehavioral responses to stress in females: tend-and-befriend,
not fight-or-flight.” In: Psychological review 107.3 (2000), p. 411 (cit. on
p. 18).
[Tro+03] W. M. Troxel, K. A. Matthews, J. T. Bromberger, K. Sutton-Tyrrell. “Chronic
stress burden, discrimination, and subclinical carotid artery disease in
African American and Caucasian women.” In: Health Psychology 22.3
(2003), p. 300 (cit. on p. 18).
[Tsi+07] P. Tsiamyrtzis, J. Dowdall, D. Shastri, I. Pavlidis, M. Frank, P Ekman.
“Imaging facial physiology for the detection of deceit.” In: International
Journal of Computer Vision 71.2 (2007), pp. 197–214 (cit. on p. 34).
[Vai00] B. G. Vainer. “Narrow spectral range infrared thermography in the vicinity
of 3 µm operating wavelength.” In: Quantitative InfraRed Thermography 5
(2000), pp. 18–21 (cit. on p. 12).
[Vai05] B. G. Vainer. “FPA-based infrared thermography as applied to the study of
cutaneous perspiration and stimulated vascular response in humans.” In:
Physics in Medicine and biology 50.23 (2005), R63 (cit. on p. 12).
[YTT16] H. Yasufuku, T. Terada, M. Tsukamoto. “A Lifelog System for Detecting
Psychological Stress with Glass-equipped Temperature Sensors.” In: Pro-
ceedings of the 7th Augmented Human International Conference 2016. ACM.
2016, p. 8 (cit. on pp. 38, 39).
[ZD04] Z. Zhao, M. Davis. “Fear-potentiated startle in rats is mediated by neurons
in the deep layers of the superior colliculus/deep mesencephalic nucleus
of the rostral midbrain through the glutamate non-NMDA receptors.” In:
The Journal of neuroscience 24.46 (2004), pp. 10326–10334 (cit. on p. 20).
[ZLA04] Q. Zhou, Z. Li, J. K. Aggarwal. “Boundary extraction in thermal images
by edge map.” In: Proceedings of the 2004 ACM symposium on Applied
computing. ACM. 2004, pp. 254–258 (cit. on p. 11).
79
[ZTP07] Z. Zhu, P. Tsiamyrtzis, I. Pavlidis. “Forehead thermal signature extraction
in lie detection.” In: (2007), pp. 243–246 (cit. on p. 20).
[Ziv04] Z. Zivkovic. “Improved adaptive Gaussian mixture model for background
subtraction.” In: Pattern Recognition, 2004. ICPR 2004. Proceedings of the
17th International Conference on. Vol. 2. IEEE. 2004, pp. 28–31 (cit. on
p. 12).
[sta16] L. statistics. https://statistics.laerd.com/statistical-guides/one-way-anova-
statistical-guide.php. Laerd statistics, 2016 (cit. on p. 52).
[the16a] O. infrared thermometers. INFRARED CAMERAS The most portable infrared
online cameras in the world. Optris, 2016 (cit. on p. 41).
[the16b] O. infrared thermometers. Interprocess Communication DLL Description.
Optris, 2016 (cit. on pp. 42, 44).
[the16c] O. infrared thermometers. http://www.optris.com/. Optris, 2016 (cit. on
p. 41).
All links were last followed on third of October 2016.
Declaration
I hereby declare that the work presented in this thesis is
entirely my own and that I did not use any other sources
and references than the listed ones. I have marked all
direct or indirect statements from other sources con-
tained therein as quotations. Neither this work nor
significant parts of it were part of another examination
procedure. I have not published this work in whole or
in part before. The electronic copy is consistent with all
submitted copies.
place, date, signature