A methodology for psycho-biological assessment of stress in software engineering

Definition of stress

Stress has been viewed from medical, psychological and organizational angles resulting in many definitions. The most general definition of stress is the general adaptation syndrome (GAS) defined by Selye (1946). The GAS definition can fit every scenario from personal short-time stress event to global long-term scenarios, and it was based on a formulation by Weinert (2004) with an organizational and workforce psychology view.

In the following we focus more closely to Weinert’s explanation of GAS, as it does not dive as deep as Selye into medical details and, hence, is easier to understand for an audience without medical background.

Stress is “… an adaptive reaction to exceeding mental or physical demands of the surroundings. Adaptive, because the resilience towards those demands is individually different.” (Weinert, 2004). Weinert (2004) derives that definition from these factors of stress:

1. Stress needs a physical or psychosocial trigger event.

2. Individuals react differently to that trigger event.

3. Constraints and demands are linked to the build-up of stress. Constraints and demands are, for example, deadlines or quality requirements connected to the trigger.

However, this does not mean that every event that fits the above definition necessarily affects a person.

In this context, commonly mentioned conditions for people to be affected by stress are (Weinert, 2004):

• The outcome or consequences of the trigger event must not be known beforehand. If the result of the stressful event is perceived as “unchangeable” it will generate no or much less stress compared to the same event with an uncertain outcome.

• The outcome or consequences of the stressor have to have an influence, either good or bad. This becomes most obvious in high-stake scenarios, for example at war, were for example, Erwin Rommel said: “Don’t fight a battle if you don’t gain anything by winning.” (Rommel & Pimlott, 2014).

For better understanding, let us make an example related to software development:

• A software product is due to be released to an important customer. The future of the company depends on this commercial operation because funds are running out. It is not known yet whether or not the customer will buy the product in the current state.

• The trigger event in our example is the prompt release of the product (deadline). The consequences of the stressor are not known, as it is not clear if the product will be sold, at what price and if this sale will keep the company financially afloat. The outcome is important to the developer because his/her job might depend on it.

• Each person in the company might experience a different level of stress connected to this scenario based, for example, on their individual judgement of the ease of finding a new job if the project is not successful and the company goes bankrupt. If a person has already taken mental dismissal and is sure to find a new job or already has a new job he/she might not experience any stress at all.

The GAS can be used to model every stress interaction in general. As in the example above, it can be used to view the impact of stress (the critical release of the software product) on the company as a whole. A more narrow focus is the theory of Lazarus (1966) which refines the idea of Selye.

By narrowing the reaction to a stressor to the cognitive processes active when dealing with stress (transactional or cognitive stress theory), this model is closer to the situation of knowledge workers such as software developers than the generalized model of Selye. In the transactional model, if a situation is assessed to be a strain, the situation can be considered as already harming, potentially harming (threatening) or as a risky, but worthwhile, challenge. The assessment and the progress of the situation based on the personal resources and the possible solutions for coping with the stressor can change over time. Based on these possibilities an actual reaction is provoked. This reaction to the stressor is, in the best case, a direct action to solve the stressful situation (fight) or, in less favorable constellations, evasion (flight) of the situation or other coping strategies (e.g., trivialization).

This cycle of assessment based on the changing surroundings and continuous evaluation of coping strategies will be repeated until an appropriate coping strategy is found, which ends the stressing encounter. If no fitting coping strategy can be found, the person is either blocked by the continuing search cycle or through the application of unfitting solutions, with a high usage of resources, the problem is gradually eroded until it can be completely overcome.

Let us again imagine a software engineering example for this stress model:

• A new member of a development team has been assigned to the first task in the project. It is a non-trivial part of the system to be implemented and it is the developer’s chance to prove his/her worth to the team.

• After the situation was found to be stressful, a second assessment of the problem reveals that the new member feels a lack of skills needed to finish the given task properly (situation assessed to be threatening). Despite that feeling he/she still starts working on the problem (coping). The new team member is working on the problem and his/her knowledge and skills increase as he/she is going and the assessment of the situation may change to “risky but worthwhile”. The assessment can change again in the course of action, for example if the new member encounters a problem which can not be fixed easily. The assessment can then again rise to threatening or even harmful, depending on the personal resources available. Still the assessment will continue until the situation is solved one way or another.

It is important to keep these definitions of stress in mind when designing a study or experiment which aims to measure stress because we need to be aware of the stress generation, which is still not fully understood (Noack et al., 2019). Most of the time we will already have a stressor we want to take a look at (e.g., project deadlines), but to keep that stressor as free from other influences as possible, and for a correct interpretation of the results later on, we need to look at potential constraints and demands which might not affect all participants in the same way. We also have to find a way to make the participant care about the outcome of the stressor if it is an artificially created stressor. There are some commonly used ways to induce stress in experiments like the Trier Social Stress Test (Kirschbaum, 2015) which uses social evaluation and cognitive demands to generate stress. Another way can be to create a competitive scenario in which participants can earn a reward based on their results on a task compared to the other participants.

Effects of stress

To understand the ways to measure stress, we need to know the basic reactions of the body and mind to stress. A frequently cited summary of the general effects of stress has been written by Kaufmann, Pornschlegel & Udris (1982). Somatic psychological short-time effects include increased heart rate, raised blood pressure and adrenalin discharge. The personal psychological effects might include a feeling of strain, frustration, anger, fatigue, monotony and saturation. The individual behavior can suffer from performance fluctuation, decline of concentration ability and bad sensorimotor coordination, leading to an increased error rate. Medium and long term stress exposure can lead to psychological problems such as dissatisfaction, resignation and depression and general psychosomatic ailment and disease. The negative effect of medium to long-term exposure to stress on the behavior can include increased consumption of nicotine, alcohol or drugs as well as absenteeism (sick days) on individual level and conflicts, quarrels, general aggression against others and withdrawal (isolation) at and outside of work on a social level. Even short-term stress can lead to unpleasant side effects, which are especially harmful for communication and cooperation-dependent work like software development. Since 1982 when Kaufmann, Pornschlegel & Udris (1982) summarized the effects, more research has been conducted supporting and extending the understanding of stress effects mentioned by them. We will go into more detail below.

Psychological measurements

Many different variations of psychological measurements of stress have been discussed over time (Cohen, Kamarck & Mermelstein, 1983; Peacock & Wong, 1990; Kanner et al., 1981). Psychological measurements can be grouped into two classes: those which assess long-term stress (e.g., Life Changing Events by Rahe (1977)) and short-term stress (e.g., Emotional Self Rating by Schneider et al. (1994)) based on self report and concrete situations, and those which look at more global coping strategies of participant who are not directly connected to a specific trigger (e.g., Stressverarbeitungsfragebogen by Janke, Erdmann & Boucsein (1984)).

The long-term assessment strategies are used to investigate the impact of stress on the well-being or health of individuals. This method uses interviews or tests that should represent the overall picture of the stressful events in someone’s life, for example, the death of a loved one or job loss.

The short-term assessment strategies try to assess the stress experienced for some minutes up to an entire day by at least two questionnaires or interviews, ideally before and after the stressful event. The items used in these tests range from a plain assessment by the participant on the momentary stress level on a scale from 1 to 5 to more episodic tales of events including the points of why this event was found to be stressing, how severe and long the stress was felt. This kind of data collection is used regularly in psychology as well as medical research and is commonly considered as reliable given good psychometric properties.

An example from the software engineering point of view to distinguish between long and short-term term stress assessment could be the difference between keeping track of a whole development project, taking a sample each day and assessing the stress of a single 4-h programing session.

The other class, aiming at looking at stress coping strategies, uses questionnaires that ask general questions like “How do you handle stressful situations?” or “What kind of situations do stress you?”. This can also be done over a period of time, to gather a stress profile, by repeating these question over the day.

Stress also affects concentration and memory (Weinert, 2004) because of the cognitive resources needed to cope with the increased stress. This use of cognitive resources is called cognitive load. Cognitive load can be measured by testing concentration and memory of participants using tests like the N-Back-test (Gevins & Cutillo, 1993).

Stress can also have a negative effect on a person’s mood, leading to frustration and anger at short-term and, at worst case, to a depression at long-term exposure, as mentioned in “Effects of Stress”. Mood can be assessed by questionnaires like PANAS (Positive and Negative Affect Scale, Watson, Clark & Tellegen (1988)) or ESR (Emotional Self Rating scale, Schneider et al. (1994)). All these questionnaire based approaches can suffer from common issues and biases when working with humans participants (King & Bruner, 2000; Paulhus, 1991; Schwarz, 1999):

• Honesty/Image management: The problem how the participant wants to present him/herself and what he/she is willing to reveal about him/herself.

• Social desirability: The distortion of answers by the participants because they feel like some answers are more socially acceptable than others.

• Introspective ability: Concerns to which degree the participant is able to understand him/herself.

• Understanding: As human language is not perfect and needs to be interpreted, formulations can be understood differently.

• Rating scales: The nuances of ratings can be interpreted differently.

• Response bias: The individual’s tendency to respond in a certain way, regardless of the actual evidence they are assessing.

Psychological tests have ways to balance such issues by providing control answers which show contradictions. Despite that, it is always desirable to back up these results with biological factors which are much harder to be influenced by involuntary effects and biases.

Biological markers

The biological factors available to us and mentioned in literature to measure stress are heart frequency, blood pressure, electrical conductivity of the skin, activity of muscles and hormone levels (Birbaumer & Schmidt, 2010). The assessment of these factors is not easy, as the difference between a normal phase of strain and abnormal stress needs to be considered and not every factor is applicable to each form of perceived stress (physiological, emotional, mental).

Measurement tools

We propose to assess stress, emotional state and cognitive load of participants, where the first factor is observed both from a psychological and a biological perspective. Figure 1 describes in greater detail the constructs under study and the related measurement instruments.

We assessed stress using biological markers through saliva samples. In particular, we measured cortisol and alphaamylase, as the saliva samples are easy to gather and the hormone/enzyme levels in it are easy to measure by any laboratory specialized in hormone analyses. We propose PANAS (Positive and Negative Affect Scale, Watson, Clark & Tellegen (1988)) and ESR (Emotional Self Rating scale, Schneider et al. (1994)) for the psychological stress and emotional stress markers as self-ratings have a good time-to-information ratio. PANAS and ESR are used to assess the participants emotional state and current mood, as increase in stress has a negative effect on mood (see “Effects of Stress”). We extended PANAS with questions asking about pre-existing stress levels and possible previous knowledge and skills related to the task to be done in the experiment, because previous knowledge can have an impact on coping strategies of the individual and have an impact on stress generation. With these extensions our PANAS scores for the positive factors ranges from 0 to 50 and the negative factors from 0 to 55.

Cognitive load (see also “Psychological Measurements” on Psychological Measurements) is conveniently operationalized and measured by the N-Back test as implemented by the computerized PEBL Psychological Test Battery (Mueller & Piper, 2014). PEBL allows the data to be gathered automatically and exported as datasets. N-Back challenges participants to memorize letters and their position in a sequence of letters which are shown one letter at a time. With a N-Back test, participants have to press a key if a letter has been repeated n steps back. For example, for the letter sequence {X, X, X, V, X, Y, Y} and a 1-Back test, the key should be pressed at (1 for key press, 0 for no key press) {0, 1, 1, 0, 0, 0, 1}. The correct key presses for a 2-Back test would be {0, 0, 1, 0, 1, 0, 0 }. The hit/miss ratio and reaction time indicates how much cognitive capacity is left to work with. In order to reduce the learning effect when the test is used multiple times, there are variations of this test (e.g., not letters, but positions in a 3 × 3 square are to be correctly remembered).

Besides demographic data, a study should also collect control-variables that might influence the stress reaction, such as pre-existing mental conditions and medications (e.g., birth control pill). Also, as this might have an increasing effect on personal stress, we asked how satisfied the participants were with their decision of life and work situation alongside the demographic questions. The data was gathered anonymously and linked together only through an anonymous identifier.

Measurement sequence

The first step in implementing our proposed method is to decide on the frequency and placement of the cortisol and alphaamylase measurements. The decision should take into consideration that, while more measurement points in a shorter period of time will provide a more detailed picture of the stress reaction to the topic under observation, it also will affect the stress generation itself. The shortest cycle is also limited by a few parameters. Participants only have a limited ability to generate enough saliva. Too frequent disruptions might be perceived as a nuisance, increase the generated stress, and might have an additional negative effect on the topic under research or even cover the stress reaction under observation. Also, the hormone system reacts in the range of minutes to hours whereas the nervous transmission reacts in milliseconds (cortisol peeks around 15–20 min after stress onset (Kirschbaum & Hellhammer, 1994)). If the stress-inducing task is too short (less than 10 min) a second sample should be gathered to increase the chance to include the peak or other measurements (e.g., heart rate or skin conductivity) can be used, but with the impediments stated above. As cortisol has a daily cycle, the gathering of the saliva samples also needs to be planned in a way that all participants are assessed at the same time to generate comparable results. In order to add noise to the cortisol and alphaamylase measurements, it is important to instruct the participants to not consume beverages containing sugar and not to eat or smoke one hour before the saliva samples are gathered.

The times when the personal stress and emotional state are assessed have to be adjusted as well. As it takes a non-trivial amount of time to fill in the questionnaires, the personal and emotional assessment should happen at the beginning of the study to determine a measurement baseline, then at appropriate times in the course of the study and at the end of it. In the case of short periods of saliva sample measurements it is sufficient to have the personal stress levels and emotional state measured at the start and end of a task. If the topic under research contains longer breaks we advise to use the questionnaire measurement at the start and end of the breaks. For long term studies, we advise to use the self-assessment via questionnaires at least twice a day, possibly at the beginning and end of the work day. By doing so, the influence of stress generated outside the object under research can be identified and taken into account when looking at the stress reaction.

In our case, the measurement of the cognitive load happens after a substantial part of the task under research and before the questionnaires assessing the personal stress level and mood, so the cognitive load is only influenced by the task.

As with the questionnaires, for long-term studies, the N-Back should be used at the start and end of the working period.

We illustrate the entire design process in Fig. 1. It represents the process as it was derived from our literature review and consultations with colleagues from the psychological and medical fields. Participants face a short cool-down period (approx. 5 min) of no activities, for controlling purposes, during which most of their markers stabilize. We then instruct participants about the experiments goals. After that, the participants sign a consent form (not present in Fig. 1). We collect a first sample of saliva, which sets the baseline stress value of participants. After the baseline stress measurement, participants fill in demographic questionnaires. Following the demographic questionnaires, we start assessing the baseline mood and perceived stress with ESR and PANAS. Participants then face the first computerized task, that is the N-Back test, for assessing the baseline cognitive load. As the N-Back test might be stressful to participants, we take a second saliva sample to be able to monitor the stress build-up for the task under observation later. The “stress” task for the experimental and control groups can then begin.

The length and amount of the experimental task parts (Fig. 1 shows two parts) should be dictated by the decision on the saliva sample interval. In our case, we take a third saliva sample at around half of the time planned for task solving. The stress markers cortisol and alphaamylase should be peaking here as the endocrine system has had enough time to respond to the initial stress trigger of the task (cortisol peeks around 15–20 min after stress onset (Kirschbaum & Hellhammer, 1994)).

We take a fourth and final saliva sample at the end of the experimental task. Finally, participants do the computerized cognitive load test once more, to measure the available cognitive resources left after the experimental task. A final set of ratings of mood and perceived stress follows. The last two activities are inverted compared to those before the task, as the cognitive load score should not be influenced by the questions for rating mood and self-assessment.

Longer studies performed over several days have a similar design to Fig. 1, varying only in the intervals of measurements as mentioned above. If the assessment of the recovery from the stressor is of interest, a measurement of all relevant indicators (cognitive load, mood, perceived stress, hormone/enzyme levels,…) should be done 40–120 min after the stressor onset. In the remainder of this paper, we report on two studies that allowed us to implement and refine the overall research design.

Pilot study 1

The pilot study implemented the method of Fig. 1 in its entirety. We only added a set of questions aiming at self-efficacy (Bandura, 1997; Jerusalem & Schwarzer, 1999; Kogler et al., 2017) to the psychological test phases, which allowed us to assess the individual stress resilience of the participants.

For the pilot we recruited 43 volunteers from the MSc course “Software Quality Assurance and Maintenance” at the University of Stuttgart. Students were in their 2nd or 3rd semester of studies. None of them reported any medical conditions interfering with the stress measurement. We were aiming for a high number of participants even in the pilot as we wanted to evaluate how easily large groups can be assessed with our chosen methods.

To induce stress, the participants were told that a list with the results of their work on the code (number of correct fixes) would be made public, so every participant could see how well he or she did, compared to the other participants. We never delivered on our threat, for obvious ethical reasons, but the stress was induced by our statements.

We split the work phase into two parts of 25 min and 20 min, the first part was a bit longer in order to compensate for the time needed to get into the task.

To avoid learning effects, the second N-Back test used positions in squares rather than letters.

Results

Unfortunately, we faced some data loss due to failing hard drives. Boxplots in Figs. 2–4 shows the data we were able to retain. Figure 2 shows the distribution of the PANAS values. From bottom to top we have the sum of the positive factors before the task, the positive factors after the task, the negative factors before the task and the negative factors after the task. The top four plots show the difference between pre and post PANAS scores. This shows the progression of the emotional state of the participants. Figure 3 shows the number of correct responses to the N-Back stimulus in percent and Fig. 4 shows the reaction time in milliseconds to the N-Back stimulus, both pre and post the task. The values for correctness and reaction time are connected, so the participants for both values are the same.

Of the PANAS and ESR questionnaires, 10 out of 20 from the experiment group and 11 out of 23 from the control group were usable. The emotional state based on the PANAS values is calculated by summing up the positive effects and subtracting the sum of the negative effects. To see the emotional change, we calculated these values pre and post-work phase. The difference of these values indicates the emotional change. Performing a repeated measures (rm) ANOVA with time (pre, post) and valence (pos, neg) as within-subject factors and group as between-subjects factor revealed no significant effect of time (F(1,18) = 1.540, p = 0.230), a significant valence effect (F(1,18) = 4.615, p = 0.046, part-eta-sq. = 0.204) with higher values in the positive valence than negative valence, and no significant group effect (F(1,18) = 0.192, p = 0.667). Moreover, no interaction reached significance (all p > 0.151).

The data of the ESR (see also Tables 1 and 2) does also not show statistically significant group differences for the emotion (all p > 0.395) and stress (p = 0.342) ratings.

Table 1:

Pre and post-experiment task values for ESR factors, experiment group, pilot, sample size = 10.

	Anger	Disgust	Happiness	Sadness	Surprise	Fear	Stress
ESR pre experiment group
Mean	1.90	1.40	1.80	1.50	1.90	1.30	2.50
Standard deviation	0.88	0.84	0.92	1.08	0.99	0.67	0.85
Median	2.00	1.00	1.50	1.00	1.50	1.00	2.00
ESR post experiment group
Mean	2.3	1.3	1.6	1.6	1.9	1.4	2.7
Standard deviation	1.06	0.67	0.84	1.07	1.37	0.84	0.95
Median	2.5	1	1	1	1	1	3

DOI: 10.7717/peerj-cs.286/table-1

Table 2:

Pre and post-experiment task values for ESR factors, control group, pilot, sample size = 11.

	Anger	Disgust	Happiness	Sadness	Surprise	Fear	Stress
ESR pre control group
Mean	2.09	2.18	1.82	1	1.91	1.36	2.18
Standard Deviation	1.22	1.33	0.75	0	0.83	0.92	1.17
Median	2	2	2	1	2	1	2
ESR post control group
Mean	2.09	2	2	1.18	1.18	1.23	2.73
Standard Deviation	1.38	1.34	1	0.4	0.98	0.65	1.27
Median	2	1	2	1	2	1	3

DOI: 10.7717/peerj-cs.286/table-2

Table 3 reports all values we gathered for cortisol and alphaamylase. See Fig. 1 for the location of the various measurements in the study progress. The sample size refers to the actual sample size used for calculations at this step, as the lab analysis reported invalid values for some steps/participants, probably due to not enough saliva samples left as they had to rerun some of the analysis. From the numbers we can see a build up of a hormone stress response with a slightly later peak in alpahamylase for the experiment group.

Table 3:

Cortisol and alpha-amylase scores, pilot.

	Experiment	Control	Experiment	Control
	Baseline cortisol (pg/ml)		Offset cortisol (pg/ml)
Mean	6.13	7.18	5.63	7.3
Deviation	4.41	4.24	3.93	5.15
Median	4.74	7.2	4.71	6.17
Sample size	14	22	17	22
	Peak cortisol (pg/ml)		Final cortisol (pg/ml)
Mean	6.69	7.11	5.38	4.5
Deviation	4.05	5.09	3.73	2.76
Median	5.77	5.81	5.39	4.2
Sample Size	15	18	11	15
	Baseline alphaamylase		Offset alphaamylase (U/l)
Mean	44.33	55.29	57.15	87.1
Deviation	47.77	83.16	86.32	81.66
Median	19.63	18.76	20.83	50.24
Sample size	14	19	14	21
	Peak alphaamylase (U/l)		Final alphaamylase (U/l)
Mean	62.39	44.73	84.08	47.35
Deviation	78.67	56.75	106.23	54.74
Median	18	18.95	18.76	20.39
Sample size	10	17	11	13

DOI: 10.7717/peerj-cs.286/table-3

The statistical tests (Willcoxon U for α = 0.05 for cortisol (C1 = 0.29, C2 = 0.31, C3 = 0.54, C4 = 0.61), Cliff’s delta for cortisol (C1 = −0.068, C2 = −0.76, C3 = −0.3, C4 = −0.564), t-Test for α = 0.05 for alphaamylase (A1 = 0.64, A2 = 0.31, A3 = 0.54, A4 = 0.36) Cliff’s delta for alpha amylase (A1 = −0.42, A2 = −0.18, A3 = −0.57, A4 = −0.61)), reveal no significant differences between experiment and control-group. A rmANOVA revealed no significant time effect (F(3,36) = 1.096, p = 0.363), no significant group effect (F(1,12) = 0.515, p = 0.487) and no significant group*time interaction (F(3,36) = 0.278, p = 0.788). Similar to the results on cortisol, rmANOVA with alphaamylase levels indicated no significant time effect (F(3,33) = 0.490, p = 0.691), no significant group effect (F(1,11) = 0.861, p = 0.373) and no significant time*group interaction (F(3,33) = 0.443, p = 0.636). The raw data of the hormone levels can be found in the appendix in Tables A1 and A2.

The increasing effect size might imply that the differences between the control and experiment groups would grow more visible if the experiment would progress longer and if we had had more usable data points. It is also possible that a greater stress induction will lead to a more visible effect.

Due to various problems at the time of the experiment’s execution, we were only able to gather 4 usable data sets (correctness/reaction time for pre and post task) for the control group and 6 data sets for the experiment group for the N-Back test. For these data sets, the other data (PANAS, ESR and hormone/protein levels) is also available. In the pilot we see an improvement of correctness and reaction time for the experiment group (see Median in Boxplot Figs. 3 and 4), while the correctness for the control group decreases. Still, analyzing the effect of intervention on cognitive performance, the rmANOVA with the within-subject factor time (pre, post) and between-subjects factor group revealed no significant time effect (F(1,8) = 0.479, p = 0.509), no significant group effect (F(1,18) = 0.091, p = 0.770), and no significant time*group interaction (F(1,8) = 3.391, p = 0.103) emerged.

Study 2

In the design of the second study the grayed-out parts of Fig. 1 are removed. This represents our changes based on the lessons learned of the pilot. We removed the initial cool-down factor and the first saliva sample. For the sake of consistency with the figure, we still call our new first saliva sample the offset stress, but we consider that measurement step our new baseline stress. The pilot test did not suggest potential differences in the measured levels within baseline stress and between baseline and offset stress, so we assume that the psychological test and N-Back session do not induce any significant stress to software developers. As a positive side effect, this elimination of one of the saliva samples reduces the time and money needed for the laboratory analysis.

We also reduced the amount of questionnaires in the psychological test phases by removing the resilience questions. The value of these question items did not help in explaining any of the effects analyzed. During our observation and in post-experiment statements, participants seemed to be most irritated by the resilience questions.

We also changed the way we induce stress to the participants in the hope that the changed procedure would show an increased reaction. First we stimulated the participants with a hardly reachable goal for the work on the software. We told them that previous participants had done a minimum of 35 fixes for problems highlighted by FindBugs in 3 different categories and that these were the low achievers. Second, as the groups were much smaller (3–5 participants per session) we could recreate an effect similar to the TSST (Kirschbaum, 2015) effect, by simply having evaluators in the room monitoring the work of the participants and pretending to write down remarks on their notepad from time to time with muttering disapproval.

For this experiment we were able to recruit 32 participants from the bachelor study course “Introduction to Software Engineering”. The participants were randomly distributed over 7 dates, 3 for the control group and 4 for the experiment group, resulting in 17 participants in the experiment group and 15 participants in the control group. Again, as in the pilot, the experiment group used our enhanced tool, while the control group used the original tool. None of the participants reported any medical conditions interfering with the stress measurement.

Results

The results of the PANAS questions (see Boxplot in Fig. 5, missing samples either did not finish or did not fill in questionnaires fully, see “Pilot Study 1” for description of layout) show that both groups experienced a decrease in mood, but the decrease was steeper for the control group (Median for positive factors declines by 6, median for negative factors increases by 3) than for the experiment group (Median for positive factors declines by 2, median for negative factors increases by 2). Still, the range of mood change for both groups is spread over a wide spectrum, so the results are not statistically significant (Wilcox p = 0.6 for α = 0.05) and the effect size (Cliff’s Delta: −0.01) is low.

Performing a repeated measures (rm) ANOVA with time (pre, post) and valence (pos, neg) as within-subject factors and group as between-subjects factor revealed no significant time effect (F(1,30) = 0.221, p = 0.642) but a significant valence effect (F(1,30) = 17.992, p < 0.001) as well as a significant time*valence interaction (F(1,30) = 6.918, p = 0.013). However, no significant group effect (F(1,30) = 0.171, p = 0.682) as well as no significant interactions with the factor group emerged (p > 0.335 in all cases). Applying post-hoc analyses on the significant interaction demonstrated a significant increase in negative mood (post > pre, p = 0.043) and a significant decrease in positive mood (pre > post, p = 0.043). In general, positive ratings were significantly higher than negative ratings (pre: p < 0.001; post: p = 0.045).

From the results of the ESR questions (Tables 4 and 5), we learned that the control group experienced an increase in disgust and surprise compared to the experiment group. Also, both groups show an increase in self-reported stress but the standard deviation for the experiment group also increased which implicates that at least some participants experienced less stress. The repeated measures (rm) ANOVA shows a significant condition effect (F(6,180) = 25.359, p < 0.001), a trend for a time effect (F(1,6) = 3.778, p = 0.061), with higher values post than pre, and no significant group effect (F(1,30) = 0.239, p = 0.629) emerged. Moreover, a significant interaction of condition*time (F(6,180) = 3.532, p = 0.007) occurred, while no other interaction reached significant (p > 0.298 in all cases). Post-hoc tests disentangling the significant interaction revealed a significant increase in anger ratings (post > pre, p = 0.001) and a significant decrease in happiness ratings (pre > post, p = 0.012). All other comparisons remained non significant (p > 0.118).

Table 4:

Pre and post-experiment task values for ESR factors, experiment group, study 2, sample size = 9.

	Anger	Disgust	Happiness	Sadness	Surprise	Fear	Stress
ESR pre experiment group
Mean	1.11	1.22	2.44	1.22	2.67	1.44	2.33
Standard deviation	0.33	0.44	0.53	0.67	1.32	0.53	0.71
Median	1.00	1.00	2.00	1.00	3.00	1.00	2.00
ESR post experiment group
Mean	1.78	1.22	2.22	1.33	2.67	1.44	2.56
Standard deviation	1.09	0.67	1.2	0.71	1.32	0.73	1.01
Median	1	1	2	1	3	1	2

DOI: 10.7717/peerj-cs.286/table-4

Table 5:

Pre and post-experiment task values for ESR factors, control group, study 2, sample size = 15.

	Anger	Disgust	Happiness	Sadness	Surprise	Fear	Stress
ESR pre control group
Mean	1.53	1.13	2.8	1.2	2.3	1.53	2.47
Standard deviation	0.64	0.35	0.94	0.56	0.98	1.06	0.99
Median	1	1	3	2	2	1	3
ESR post control group
Mean	2.2	1.53	2.53	1.47	2.47	1.47	2.73
Standard deviation	0.94	10.6	0.74	0.74	1.19	0.64	0.96
Median	2	1	3	1	2	1	3

DOI: 10.7717/peerj-cs.286/table-5

The cognitive load (see Figs. 6 and 7, missing samples did not finish both N-Backs, see “Pilot Study 1” for description of layout) again shows the same effect as seen in the pilot (correctness (mean increased by 1.5% points) and response time improvement (median decreased by about 125 ms) for experiment group vs. correctness decrease (mean by 1% point) for control group) but still is not statistically significant.The Wilcox test does not reveal a statistical relevance (p = 0.3217, α = 0.05). The slight reduction in response time and increase in correctness for both groups originates most likely from a learning effect.

Performing the rmANOVA with time as within-subject factor and group as between-subjects factor showed a significant time effect (F(1,24) = 5.389, p = 0.029), no group effect (F(1,24) = 3.026, p = 0.095) but a significant time*group interaction (F(1,24) = 6.669, p = 0.016). Disentangling the significant time*group interaction revealed a significant group difference before the intervention (pre: p = 0.022), with better performance in the control group. After the intervention (post), no significant group difference emerged (p = 0.625). While the experimental group did not show a change in performance (pre vs. post, p = 0.517), the control group showed a significant decrease (pre vs. post, p = 0.041).

The median of the cortisol measurements (see Table 6) for the control group are slightly higher but the alphaamylase values are lower. However, this test shows no statistical significance (Willcoxon U for α = 0.05 for cortisol (C1 p = 0.35, C2 p = 0.4, C3 p = 0.68), Cliff’s delta for cortisol (C1:0.21, C2:0.19, C3:0.10), t-Test for α = 0.05 for alpha amylase (A1 p = 0.14, A2 p = 0.25, A3 p = 0.07), Cliff’s delta for alpha amylase (A1:−0.31, A2:−0.31, A3:−0.15)). The raw data of the hormone levels can be found in the appendix in Tables A3 and A4.

Table 6:

Cortisol and alpha-amylase scores, Study 2.

	Experimental	Control	Experimental	Control	Experimental	Control
	Offset cortisol (pg/ml)		Peak cortisol (pg/ml)		Final cortisol (pg/ml)
Mean	6.13	7	4.55	5	3.83	4
Deviation	3.36	2.65	2.53	2.39	2.31	2.68
Median	5.7	6.85	3.44	4.33	2.85	2.95
Sample Size	15	14	15	14	15	14
	Offset alphaamylase (U/l)		Peak alphaamylase (U/l)		Final alphaamylase (U/l)
Mean	62.99	55	84.24	75	94.12	91
Deviation	16.51	13.11	13.22	25.37	23.42	23.5
Median	62.4	53.85	87.4	67.75	100.4	93
Sample Size	15	14	15	14	15	14

DOI: 10.7717/peerj-cs.286/table-6

The rmANOVA with time and group revealed a significant time effect (F(2,54) = 21.451, p < 0.001), indicating a significant decrease from t1 to t3 (t1 vs. t2: p < 0.001; t2 vs. t3: p = 0.012; t1 vs. t3: p < 0.001), but no group effect (F(1,27) = 0.178, p = 0.676) nor group*time interaction (F(2,54) = 0.163, p = 0.810).

Similar to the cortisol analysis, the rmANOVA for the Alphaamylase indicated a significant time effect (F(2,54) = 34.361, p < 0.001), demonstrating a significant increase from t1 to t3 (t1 vs. t2: p < 0.001, t2 vs. t3: p = 0.003; t1 vs. t3: p < 0.001), but no significant group effect (F(1,27) = 1.525, p = 0.227) nor a significant group*time interaction (F(2,54) = 0.262, p = 0.771). This means that the experiment group had a lower chemical stress reaction but a higher need for chemical energy. This might be an indicator that the experiment group reached an energy consuming coping strategy for the given problem sooner. The statistical power increases which indicates, that a larger group or a greater stress induction could show a more prominent effect.

Still, the overall design proved to be feasible and changes made compared to the pilot have shortened the time needed to analyze the generated data.

Limitations

Based on lessons learned (and the useful suggestions of two anonymous reviewers), we summarize potential limitations that threat the validity of results, should our design be adopted.

Unreported medical conditions

Participants might refrain to reveal severe medical conditions, like Addison’s disease or Cushing’s syndrome, which influence the levels of hormones that are measured in our design, but we also believe that such issues have negligible impact on the results of studies from our design. First of all, conditions with severe impact on the measurements are rare, as, for example, Addison affects about 0.9 to 1.4 per 10,000 people in the developed world (Neto & De Carvalho, 2014) and Cushing’s syndrome is even rarer (Lindholm et al., 2001). Also, values arising from pathologically changed hormone values should be visible as outlier in the data.

Stress induction

The stress induced to participants has to be large enough to be significant and distinguishable from the (quite low) stress induced by the measurement instruments (N-Back, saliva samples and questionnaires). We have tried to balance stress induction vs. quality of measurement with our selected tools. Our collected data has shown that our stress induction effect has been too low. To plan the stress induction accordingly, we advise to take a closer look at the different way stress can be induced in psychological research (e.g., Kirschbaum (2015) as an example of socially induced stress or Kang & Fox (2000) as an example of cognitive stress induction).

Learning effects of participants

Some stress measurement tools (e.g., the N-Back test) base their results on memory and reaction time of the participants. To keep the effect of learning at a minimum the tests should be randomized to the best possible extent. In our case we used two different versions on the N-Back (see “Measurement Tools”).

Reuse of the stress task under research

If a task is being reused, participants might give away information about the task to other future participants. The information flow should be restricted if possible, as uncertainty is part of the stress generation (see “Definition of Stress”). While in our case the task was reused, our task consisted of several many subtasks with no clear correct answer: communication would not have done harm. It was also our design to keep participants uncertain about their performance on the task, as well as the task itself.

On effort and monetary costs

The cost per combined measurement (cortisol and alphaamylase) and basic statistical analysis for one saliva sample was about 5 Euros (Swiss Health Care, 2017, http://swisshealthcare.de). This amounts to about 20 Euros per participant for the pilot and about 15 Euros per participant for the second study. Probably this cost can be reduced by arranging a high-volume contract with a laboratory or finding a cheaper provider of this service. Also, analysis of only cortisol reduces the cost per measurement by up to more than 50%.

Sometimes, as in our case, university departments (e.g., medicine, biology, chemistry) already have a contract with a laboratory or might even be able to provide the analysis themselves.

Compared to other equipment for stress measurement, our method lies in the mid section of overall costs. There are cheap heart rate monitors and skin conductance measurement tools but it depends again on the study and the aspects to be tested if they can be used effectively.

Higher priced tools, like EEG (electroencephalography), can deliver other, maybe more precise results but the process is not easily applicable to large groups. Also, those tools need consumables, like electrodes, driving up the costs. Additionally, we need to keep in mind that with the cost for the hormone analysis equipment we are also covering a small part of the analysis of the results as well, as most labs deliver basic statistical calculations (mean, median, standard deviation,….) with the raw measurement data.

The analysis of the results of other tools mentioned will require the help of medical trained personnel; for detailed results rather than just coarse indications, while with the data provided by the lab we can analyze effects with basic statistics.

No process is worth the effort if it does not deliver results. Our case is somewhat inconclusive. We can identify times with higher and lower stress from our results of the hormone levels. The additional information gathered (e.g., the N-Back results) backs up these results. However, we were not able to find a significant difference between the groups in our studies but we strongly suppose that this is due to other problems within these experiments (e.g., too small sample size due to data loss) or that the effect we were trying to observe was too small to be observed with this kind of design. In conclusion we believe that our proposed measurement process can enable non medical or psychological researchers to examine the influence of stress in processes. Our proposed measurements allow for a more detailed analysis, quick and easy applicable even in larger groups. However, our method can only deliver a first glance at stress-related problems. For an in-depth research on stress effects we strongly recommend to seek cooperation with medical or psychological stress scientists.

Data protection

We tried to gather as few personal data as possible but our proposed method will collect sensitive data beyond the usual demographic data of software engineering studies, that is, medical data. Besides adhering to the local laws on data protection, we believe that extra care should be taken when gathering, storing, and processing these data.

Before the pilot, we had an extensive discussion with the data protection agency, a German federal agency in charge of enforcing and consulting on data protection laws. We decided to remove the personalization of the data points (pseudo-anonymised data). We talk about pseudo-anonymised data as the recent European data protection law especially has this term within its text in contrast to the old law which defined a term of “anonymised data”. Pseudo-anonymisation is reached when no relation can be easily established to the personal data (e.g., names, gender…) in contrast to full anonymisation, where the relation to personal data can be reestablished under no circumstance. In our case, we would only reestablish the connection to personal data if we were able to access to the data of all university students and, even then, only with a tiny probability. In other words, we cannot reestablish the connection.

It is even more important to supply a written statement that explains what data will be gathered, for what purpose and the right to revoke the agreement and, also, the permission to use the gathered data at any time. With his or her signature, each participant should agree to these terms beforehand. This also shows the participants that their personal data will be handled securely and fairly.

Effectiveness and ethics of stress induction techniques

The issue of putting human participants under stress despite the potential health risks has to be addressed. We believe that the risk of permanent problems as a result of an experiment as described here is highly unlikely. The stress created is only temporary and is not more severe than day-to-day stress peaks. Still, it might be desirable to screen potential participants for preexisting issues which can amplify the negative effects of stress (e.g., a mental disorder). It is also necessary to fully inform the participants about the stress parts of the experiment beforehand if possible. Additionally, we advise to contact an ethics committee, if available, especially if drastic changes to the stress induction are made. Still, to our experience a formal investigation by an ethics committee is not necessary for this kind of study.

Additionally, we believe that there is a similarity with the issues in controlled experiments observing affect (moods, emotions). As summarized by Graziotin (2016), several studies have doubted the effectiveness of short mood-induction techniques for psychological experiments, where participants’ affect is manipulated through several techniques, for example, watching a sad movie, and effective long-term induction techniques might raise several ethical concerns, for example, as with the Facebook emotion contagion experiment (Shaw, 2016). Seeing how difficult it has been for us to manipulate stress when employing a robust methodology, we wonder whether the same mechanisms occur for stress induction technique, and if we should rather perform in situ studies. However, this reasoning is speculation at this point. Future studies should address the question of whether stress induction techniques are ineffective for controlled experiments.

Appendix of raw data

Questionnaires