Evaluating Knowledge of Business Processes

Any organization relies on processes/procedures in order to organize the operations. Those processes can be explicit (e.g. textual descriptions of workflow steps or graphical descriptions) or implicit (e.g. employees have learned by experience the steps needed to ‘get things done’). A widely acknowledged fact is that processes change due to internal and/or external factors. How can managers make sure the employees know the last version of the process? The current practice is to test employees by multiple-choice questions. This paper proposes a novel knowledge-testing approach based on graphical and interactive questions. To validate our approach, we set up a single-factor controlled experiment with novices and experts in a faculty admission process. The results show that our approach has better results in terms of correct answers.


Introduction
This paper introduces our research on testing methods that can be employed in testing the knowledge of business processes.This study approaches the research question: Which method of testing the knowledge about processes/procedures is more suited for evaluating the capacity of employees to execute them?Typically, process knowledge is evaluated by asking multiple-choice or open questions on the process documentation.Most organizations document some of the main processes in textual form and then struggle to keep it up to date.A growing number of organizations acknowledge the importance of business processes, document them using some form of graphical models, and employ information systems tailored to support them.However, there are still many organizations that haven't documented processes.In these organizations, employees learn by training and/or experience how to achieve organizational goals.No matter if process documentation exists or not, it is critical for managers to be able to evaluate how well the employees are able to execute processes.Shallow knowledge about processes relates to learning the main steps to be performed, their sequence, the documents and data involved, etc.A deeper understanding resides in, for example, issues like contingency steps in case of errors; overview of organization-wide processes, etc.Current testing based on multiplechoice questionnaires don't go beyond the shallow understanding.We argue that a new testing approach is needed.It should aim to put the tested employee in the position where problem-solving deep knowledge is needed, rather than the ability to memorize process steps.In this paper we introduce a first take on this challenge.We evaluate if asking questions in a graphical-interactive manner is better than the multiple-choice way.Better is interpreted in terms of a greater number of correctly asked questions as well as the time needed to answer comprehension questions.The context of our experiment is an organization of higher education.More specifically, we use the annual admission process to evaluate how accurate different types of participants know and understand the entire process, and if they are able to execute it in any specific case that might arise.The paper unfolds as follows.First, we provide an overview of the theoretical foundations by reviewing papers related to factors influencing model understanding and experimental design.In the next section, we provide the details of our controlled experiment.The 1 DOI: 10.12948/issn14531305/20.3.2016.01single factor of the experiment is the comprehension question presentation format.In section 4 we introduce the results and our data analysis.We end with conclusions and the implications of our findings.

Related Work
Business process models are key artefacts in the development of information systems.While one of their main purposes is to facilitate communication among stakeholders, little is known about the factors that influence their comprehension by human agents.To date, the body of research on process model understanding relies on controlled experiments based on multiple-choice comprehension questions.Therefore, in this section, we approach two main related research avenues: one on process model understanding, and a second one on controlled experiments in process model settings.The notation that we use in our experiment is the Business Process Model and Notation (BPMN) which is the industry standard in process modelling.It has straight-forward syntax and semantics.So-called connectors (XOR, AND) define complex routing constraints of splits (multiple outgoing arcs) and joins or merges (multiple ingoing arcs).Through XOR, when splitting, the sequence flows to exactly one of the outgoing branches.When merging, it awaits one incoming branch to complete before triggering the outgoing flow.Through AND, when used to split the sequence flow, all outgoing branches are activated simultaneously.When merging parallel branches process flow waits for all incoming branches to complete before triggering the outgoing flow.

Process Model Understanding
Process models typically capture in some graphical notation the tasks, events, states, and control flow logic that together constitute a business process.The understanding of the process is essential when it comes to achieving organizational goals, or when just passing information to third parties.Ignorance of the procedures/processes can result in either unsuccessful achievement of some goal, or in wasted resources (e.g.employee extra workload to correct mistakes).There is no clear definition of the notion of understanding of business process models.It is used in different manners depending on the context [1], [2], [3], [4].Even though a definition lacks, this notion is very important.In the academic setting, the ability of humans to understand processes is linked to model features such as structuredness, complexity (e.g.number of model elements, number of types of model elements), or the particular modeling notation (e.g.formalism needed to depict the model, how the model is actually drawn, if it follows secondary notation conventions, etc.).In organizations, the domain knowledge also plays an important role.After all, a model of which all aspects are understood very well by the stakeholders is easily verifiable from the validity and completeness points of view.In state-of-the-art research in business process management there is a clear stream of research aimed at clarifying what makes process models understandable.This stream of research is divided into 3 main branches: a) research focused on model characteristics.
Basically, researchers set out to find out what makes process models complex.As complexity is directly linked to understanding, it was proven that a model with e.g. more elements, more crossing arcs, less structure, etc., will increase the cognitive load and thus reduce the comprehension performance.What is worth mentioning, in the context of our project, is that all this work relies on similar setups of controlled experiments.Basically, multiple choice comprehension questions are asked on process models and independent variables such as various model metrics (e.g.number of model elements, number of model arcs, etc.) are linked to dependent variables such as the number of correct answers or the time needed to answer the questions.Of course, many variations can be found, linked to the specific purpose of the research (e.g.model activities with textual or abstract labels, artificial or industry models, etc.).b) research focused on the process of creating DOI: 10.12948/issn14531305/20.3.2016.01models.This kind of research aims to bring further light into the best practices of creating process models.Researchers also rely on controlled experiments, but this time aiming to capture the flow of modeling activities.For example, experimenters record how novices and experts perform the modeling activities, being given identical process descriptions.c) research focused on the connection between process models and real life implementations.This kind of research leaves the abstract approach to a more real-world focus.Thus, the research method of choice are case studies.

Controlled Experiments in Business Process Management (BPM)
We designed our study according to the research methodology laid out by Field and Hole [5] as well as Creswell [6].Shortly, we followed the recommended stages of research: Planning (i.e.literature research for related papers; choose the method for empirical research; and design the controlled single-factor experiment), Execution of the experiment, and Data analysis and interpretation.
For the literature research we performed the first steps towards a systematic literature review.We searched Google Scholar for a combination of key words (i.e."experiment business process" and "experiment process model") in order to extract all papers related to experiments in BPM area.We considered all the hits on the first 15 pages and screened the titles for relevance.The papers passing this first filter were filtered again based on their abstracts.One last filter was based on the number of citations and the year of publication (i.e.we divided the number of citations to the count of years since the paper was published and retained as relevant the hits with a ration of at least 5).Out of the third filter emerged a total of 7 papers that very closely related to our own effort.Key researchers in the area appear to be B Webber, M Weske, J Mendling and H Reijers.The papers that provided the most inspiration were [77], [8], [9], [10], [Error!Reference source not found.], [Error! Reference source not found.].
In choosing the research design, appropriate for our particular research problem, we considered the following:  the means of obtaining the information;  the availability and skills of the researchers;  justification of the way in which selected means of obtaining information will be organized, and the reasoning leading to the selection;  the time available for research.Execution of the project is a very important step in the research process.If the execution of the project proceeds as planned, the collected data is adequate and dependable.A major concern was ensuring that the survey is under statistical control so the collected information is in accordance with the pre-defined standard of accuracy.After the data was collected, we turned to the task of analyzing them.The analysis of data comprises a number of closely related operations such as: creation of categories, the application of these categories to raw data through coding, tabulation and drawing statistical inferences.Coding operation is usually done at this stage through which the categories of data are transformed into symbols that may be tabulated and counted.Editing is the procedure that improves the quality of the data for coding.Tabulation is a part of the technical procedure where the classified data is put in tables.By statistical tests we seek to test whether observed differences are real or the result of random fluctuations.After analyzing data, as briefly described before, we moved to hypotheses testing.Do the facts support the hypotheses or they happen to be contrary?This is the usual question which should be answered while testing hypotheses.Various tests, such as Chi square test, t-test, F-test, ANOVA, etc. have been developed by statisticians for the purpose.Hypothesis-testing will result in either accepting the hypothesis or in rejecting it.If a hypothesis is tested and upheld several times, it may be possible to arrive at generalizations, i.e., to build a theory.DOI: 10.12948/issn14531305/20.3.2016.01

Experiment Overview
As understandability is a rather broad aspect and cannot be measured directly, we rely on a controlled experiment to gain insights into the research question.The goal is to investigate the impact of the presentation in an effort to answer the main research question: Which method of testing the knowledge about processes/procedures best reflects the ability of experts to perform these processes/ procedures?We experiment with one factor: how the knowledge test questions are formulated and presented to the subject.Basically, we ask the same basic test question but we introduce/show it to the user in two different ways.There are two levels of this factor: a) a classic multiple choice questions layout and b) a cus-tomized graphical interface tailored for process-related knowledge.Below, we introduce one example of such a comprehension question, with its two presentation variations for the comprehension question: "Indicate the minimum number of steps to confirm your place within the admission".The participants in the experiment were 16 faculty staff and students.All had expertise with the faculty admission process that was used as a setting for our study.Participants were divided into two groups: experts were designated faculty staff that were involved in at least 3 executions of the process while novices were designated participants involved in at most 2 executions of the process.Participants were randomly assigned to one of the two factor levels such that we maintain an even distribution of experts and novices in the two groups.DOI: 10.12948/issn14531305/20.3.2016.01 The objects of the experiment are comprehension questions.Participants answered a set of ten questions about the business model, its documents, tasks, data objects and exception situations.
We use two dependent variables in the experiment.First variable is the correctness in answering comprehension questions.It is coded 1 if the correct answer was indicated by the participant, and 0 otherwise.The second dependent variable is the time needed to answer each comprehension question.The difficulty of answering comprehension questions can be linked to the amount of time it took for people to provide their answer.Time was recorded manually from the moment the question was introduced to the participant until the answer was given.Time is stored in seconds.
We also use several independent variables like: a) experience -codes the domain knowledge with the faculty admission process.There are two possible values: 0 for novices and 1 for experts, b) type of questionstores the focus of the comprehension question (e.g. can be sequence of process activities, documents and/or data needed to execute the process activities).In total, there are 4 question types, c) treatmentcodes the factor level the participants were assigned to.Basically, we formulate six hypotheses about the relationship between independent and dependent variables:  H1 The presentation format of the comprehension questions (i.e.treatment) will impact on the ability of the participants to provide the correct answer (i.e.correctness),  H2 The general process knowledge of the participant (i.e.experience) will impact on the ability of the participants to provide the correct answer (i.e.correctness),  H3 Comprehension questions on different process perspectives like control-flow, data (i.e.question type) will impact on the ability of the participants to provide the correct answer (i.e.correctness),  H4 The presentation format of the comprehension questions (i.e.treatment) will impact on how much time a participant needs to provide an answer (i.e.response time),  H5 The general process knowledge of the participant (i.e.experience) will impact on how much time a participant needs to provide an answer (i.e.response time),  H6 Comprehension questions on different process perspectives like control-flow, data (i.e.question type) will impact on how much time a participant needs to provide an answer (i.e.response time).
Personal factors have also been recognized as important factors for this type of research.They relate to the reader of such a model, for example with respect to one's educational background or the perceptions that are held about a process model.The way information is processed by humans is influenced by cognitive styles, which can be related to personality.There are persons who prefer verbal over visual information.From this point of view, through visualization, perceptional capabilities of a person are also relevant.These capabilities differ between persons with different modeling expertise [1].A level of professional expertise is assumed to take at least 1,000 to 5,000 hours of continuous training [13].Such regular training is needed to build up experience and knowledge regarding a specific process.We are unable, and thus did not aim, to capture such personal features as independent variables.

Tasks
As mentioned before, we investigate the impact of the presentation on process knowledge testing.To this end, we created a questionnaire that tests how well participants understand the process, from various perspectives.A questionnaire is a research instrument consisting of a series of questions and other prompts for the purpose of gathering information from respondents.One variant of the questionnaire consists of multiple choice questions on the admission process.The other variant implements our own proposal of question format.DOI: 10.12948/issn14531305/20.3.2016.01 The questionnaire was developed by iterating through three steps.First, we collected the textual description of the admission process and screened for issues that might rise comprehension problems.Our concern was to cover different types of understanding perspectives.For example, one major concern of managers is that employees know the correct sequence of the steps that need to be performed.Another concern is that employees know exactly which documents need to be requested at specific points in the process, etc.Second, we formulated questions based on those issues and 3 to 5 answer options.The questionnaire contains the same questions, with the same answer options, but presented in two different styles.Therefore, the third step was to create the two presentation variants in such a way to preserve neutrality regarding 'guessing' the correct answer.The questions which are included in the questionnaire are: 1. Specify which documents are necessary to enroll to the faculty.2.You have been admitted!Specify the minimum number of steps for confirming the enrollment.3. It is mandatory for a student to take the Lingua exam? 4. I'm Ionescu, a student from Switzerland.
Which is the previous step that I need to go through, before I submit my entry file? 5. What happens if I don't submit the original Baccalaureate degree for the confirmation?6.It is mandatory to collect the first downpayment when I register?7. Specify which documents are necessary to be included in the confirmation file.8. I'm Ionescu.It is mandatory to enter data in the online pre-registration system? 9. I'm Ionescu.I wasn't assigned a place neither the admission average nor depending on the option.What happens after the initial distribution?10.Online enrollment need to be made before or after the preparation of the documents from the file?
Questions are grouped in different types, such as:  Two questions related to documents.The participants are tested if they know what should be include in the student's brief;  Questions related to the control-flow.
There are three questions which inquire about the sequence (i.e.order in which activities should be performed).Such a question asks the participant if he could do something without doing something else previously.For assessing knowledge on concurrency, there is one question that inquires about the order in which some actions can be executed.Finally there is one question related to process overview, which asks the participant to indicate the smallest number of activities to be executed between two process points;  Questions related to decision making.
There are three questions which highlight the alternative way that may be followed when a decision needs to be made or when an error occurs.The collected data was be analyzed using statistical methods to verify the degree of correlation between participant's perception of processes knowledge and the proposed metrics.While the multiple choice items were evaluated automatically, the open answers had to be interpreted and matched with the errors detected based on the textual description.

Subjects
Overall, the experiment was performed on two groups made of faculty staff and students of the Faculty of Economics and Business Administration in the Babes-Bolyai University of Cluj-Napoca.All participants had a direct involvement in the execution of the admission process.The experts have more experience with the back-office side the admission process.Students have better experience with the front-office side of the process.However, all participants should have a good understanding of the processes activities and, even more, were provided with the by-law which gives the official textual description of the process.Each participant was randomly assigned to a group that received one treatment.The first DOI: 10.12948/issn14531305/20.3.2016.01 group was given the multiple choice version and the second group had the graphic interface version.

Method and Experimental Design
The overall phases of this research project are: 1. Documentary research; 2. Development of the solution and implementation of the prototype system; 3. Experimental validation by controlled experiment; 4. Analysis of experimental data; 5. Dissemination of the result.We chose to perform a controlled experiment, in which all factors remain fixed except one.The steps of the experiment execution were: 1) the experiment begins by randomly assigning the participant to one of the two treatments; 2) the operator gives the participant a brief description of the experiment goals, its setting and the tasks to be performed.
3) the document with the official procedure is provided to the participant and the operator explains that it can be consulted at any moment during the experiment.A 15 minutes interval was granted to the participant for refreshing their memory based on this document; 4) the operator introduces each question and the answer options.A timestamp is recorded; 5) the operator records the answer option indicated by the participant and a second timestamp.

Instruments
According to the theoretical background, both the characteristics of the reader of a process model and those of the process itself impacts on the understanding that may gain from studying that model.
The format and content of the questions were developed and tested in several iterations, before the final version of the questionnaire was reached.The questionnaire implementation was done in Balsamiq Mockups.Data collection was done using Microsoft Excel.Then it was imported for analysis in the statistical package MedCalc.

Research Results
In this section we introduce the results of our experiment.The collected data includes information such as: the first name, last name, treatment, question number, the answer option indicated by the participant and the task execution time measured in seconds.Additional data was added such as: the correctness of the answer option (i.e. 1 for a correct answer and 0 for incorrect one), expertise (i.e.0 for novices and 1 for experts), and question type (e.g.questions asking for sequence were coded 1, questions testing for process document knowledge were coded 2, and decision-making questions were coded 3).For each question asked to a participant one row was created.A pre-processing step was performed by manual inspection of the data for obvious errors or outliers.No abnormal observations were detected, thus all 160 observations were used for further analysis In the second stage, the results were analyzed using the MedCalc statistical package, which implements:  Descriptive data analysis ( e.g. the average of questions answered correctly according to the model and to the type of subjects, average time spent per filled in question according to the model and to the type of subjects.); Correlation analysis;  Variation analysis (e.g.ANOVA, AN-COVA, etc.).Looking at Table 2, the obvious insight is the weak, but significant correlation between Treatment and Correctness.This supports our main hypothesis that the presentation of the comprehension questions is linked to how many correct answers are provided by the participants.The positive value of the correlation needs to be read in the context that Treatment was coded 0 for the 'classic' multiple choice and questions and 1 for the graphical interface while Correctness was coded 0 for an incorrect answer and 1 for a correct answer.
There is a weak negative correlation close to the significance threshold (p=0.07) between experience and time.This points to the conclusion that experts spend less time in answering comprehension questions.
Having some interesting correlations between variables, we further analyzed the data to test out hypotheses.The analysis of variance groups the recorded Correctness by the Treatment codes.Our proposed graphical approach to asking the comprehension questions was coded 1 while the 'classical' multiple choice questions were coded 0. As ANOVA shows, the averages of the two groups are significantly different (p<0.001),while out proposed approach has a much higher correct response rate than the classic multiple choice group (59% versus 27%).Our main hypothesis, H1, is supported.It relates to the research question of this paper, that an interactive graphical approach to asking comprehension questions leads to a greater number of correct answers, which in turn are linked to a better understanding of the process.We believe that this increase in performance can be attributed to a better understanding of the questions and their context.The only other hypothesis partially supported is the obvious influence of experience over the time needed to answer the comprehension questions.The weak correlation can be attributed to the fact that we asked questions about specific, rarely occurring issues in the process.Also, the answer options were formulated in such a way that the participant could not 'guess' the answer.Unsurprisingly, the null hypothesis of H2 finds support.Experience does not seem to imply a better performance.We believe that this is linked to our careful formulation of the comprehension questions in such a way that there would be no obviously 'right' answer.DOI: 10.12948/issn14531305/20.3.2016.01

Conclusions. Threats to validity
In this paper we investigated which method of testing the knowledge about processes is more suited for evaluating the capacity of experts to execute them.The empirical validation supports our claim that a better comprehension performance is achieved through graphical interface than the multiple choice.We believe that this increase in performance is closer related to the actual level of understanding of the process by the participants, as indicated by the results of the ANOVA in Table 3 that point to a very low average of correct answers in the 'classical' multiple choice group.This conclusion is further strengthened by the fact that experience in executing the process doesn't impact on the correct answer rate, thus confirming that we formulated questions that cannot be simply answered by 'guessing' or from general knowledge.We acknowledge there are many threats to the validity of the results of our study, and tried to mitigate them.Conclusion validity of this study is limited by the sample size of collected data (i.e. 2 treatments, 1 process, 10 questions, 16 respondents).One particular aspect of the external validity of the presented research relates to the limited number of respondents from a single organization.Construct validity is linked to how the dependent variables (correctness, time) were linked to the data collected from the questionnaire filled in by the subjects.The measurements may have lacked accuracy, given that we used a manual recording of the times at which participant started and completed one task.To reduce this threat, respondents received detailed instructions about how to use and fill in the questionnaire and experimental sessions were performed with a single participant.Regarding Internal Validity, we considered several aspects that may have threatened it: -Persistence Effects.The experiment was executed by participants who had never done a similar experiment before.-Knowledge of the Universe of Discourse.
The knowledge of the domain did not affect internal validity, since it was familiar to all subjects.-Fatigue Effects.The total experiment time for each participant was less than 30 minutes, thus fatigue effects were unlikely to appear.-Subject Motivation.The subjects were highly implied to this research and the results could be a beneficial to them since the experts are directly implied in the admission process and the novices study at this university.-Plagiarism and Influence Among and Between Subjects doesn't exist, because they haven't seen each other, so they can't inspire from the colleague.
Our conclusions impact on the industry.We believe that there is enough support for moving to the next phase of implementing the proposed interactive graphical approach to testing process knowledge in the form of an application targeted at employees of organization where procedure understanding is critical.Such an application will also enable us to perform the experiment on a larger sample of participants and on multiple processes, and thus reinforce our conclusions.

Table 2 .
Correlation table

Table 3 .
One-way ANOVA for Treatment and Correctness variables

Table 4 .
Table 4 summarizes our findings: Results of hypotheses testing