Contribution of Fuzzy Logic to Evaluation

Evaluation is an important subject for medical informatics domain. The investors and managers need to know the success level and poor sides of their information system to make improvements. There are many evaluation frameworks proposed for healthcare in the literature. Although healthcare is very suitable, none of the existing evaluation frameworks employ fuzzy logic methodologies. Our proposed expectation based evaluation framework in the previous work for hospital information systems is used for examining the difference and contribution of fuzzy logic use in evaluation. The results of the framework are recomputed both by crisp computation method. The difference between fuzzy and crisp computation is examined. The study shows that use of fuzzy logic makes a difference. Eight (8) of the 17 variables appeared to have statistically significant difference. Using crisp values in evaluation may result in loss of precision. We believe that fuzzy logic helps to obtain a more realistic evaluation by taking blurred boundaries into consideration.


Introduction
Hospital Information Systems (HIS) are employed in hospitals for providing healthcare professionals with the qualified computer support they need.Fully integrated HIS' are highly costly projects.Deploying these huge systems has great risks.The big one is the risk of failure.Literature says nearly 70% of these implementation projects fail (Ammenwerth , Gräber , Herrmann , Bürkle , & König ,2003) .This finding makes the investors and the managers question the "goodness" of their system.To measure the "goodness," system evaluations are made.In the literature many different definitions of evaluation are available.Briefly, we defined the evaluation, by drawing from the literature, as "measuring the extent of meeting the specified criteria of a system, in a specified context" (Gürsel , Zayim , Gülkesen , Arifoğlu , & Saka , 2014).International Atlas of Evaluation tells that the number of evaluations is rapidly increasing (Furubo , Rist , & Sandahl ,2002).These evaluations are made both by government and public sector organizations (Furubo , Rist , & Sandahl ,2002).
The importance of evaluation studies for HIS can be summarized by the word "You can't manage it, if you can't measure it" (Protti, 2002).To improve HIS, it must/should be evaluated from the time being started to be developed, to the time taken out of operation, i.e. in the system's life cycle, iteratively (Al-Yaseen , Al-Jaghoub , Al-Shorbaji , & Salim M, 2010, Ammenwerth , Iller, & Mahler, 2006;Protti, 2002).HIS evaluation also helps eliminate implementation problems by means of on-time interventions (Kushniruk, & Patel ,2004).The evaluation process is considerably complex and this issue is more recognized in Medical Informatics than any other domain.All we know healthcare domain itself is a huge combination of problems.When we put the words "Evaluation" and "Medical Domain" together, then the "complexity coefficient" increases exponentially.The complexity of the medical domain may stem from the vagueness nature of it.In medical science, it is impossible to give exact definitions or descriptions of medical concepts and relationships between concepts in most of the cases.The boundaries are not clear.Consider the statement "If the back pain is severe, and the patient is old, then apply acupuncture to certain point for a long time."To process this statement in a computer system, we need more than programming skills.All the terms we need to model; severe, old, certain point, long time, are vague and fuzzy.For that reason HIS applications employ fuzzy logic methodologies.
Fuzzy logic is presented to the literature by Zadeh (Zadeh, 1965).But Dadone has stated in his PHD work that although fuzzy logic the concept of multi value logic was first introduced in 1965 to handle vagueness, it was present in the beginning of the century (Dadone , 2001).He gives some sayings to validate his claim such as; Pierce "I have worked out the logic of vagueness with something like completeness" in 1905.
Fuzzy logic is used when there is uncertainty and vagueness in the situation.Consider the five point Likert scale, used in this study, (very important, important, average important, not so important, not important) one cannot know where very important ends and where important begins.Likewise one cannot know where important ends and where average important begins and so on.Because of this reason, it may not be true to assign crisp values to them in computations.That may cause loss of precision.They should be represented as a range of fuzzy numbers instead of crisp values.Fuzzification, fuzzy operations, defuzzification are the basic fuzzy logic methodologies.A linguistic variable must be converted into fuzzy numbers (fuzzification) to make fuzzy operations, which are fuzzy addition, fuzzy subtraction, fuzzy division and fuzzy multiplication.The resulting values of the fuzzy operations are again a fuzzy number that has meaning in the real world by itself.So it needs to be converted into crisp numbers, which is the defuzzification.

Background
Many HIS evaluation frameworks were proposed in the literature (Ammenwerth , Gräber , Herrmann , Bürkle , & König ,2003;Dixon ,2006;Goodhue, Klein , & March ,2008;Heeks ,2006;Kazanjian , & Green , 2002;Shaw ,2002;Yusof , Kuljis, Papzafeiropoulou , & Stergioulas,2008).As mentioned in the definition, evaluation is context dependent, so different frameworks are proposed for different contexts and purposes.In the previous work, we proposed a new evaluation framework in the user expectations context (Gürsel , Zayim , Gülkesen , Arifoğlu , & Saka , 2014).The purpose was to evaluate HIS' in measuring the extent that it meets users' expectations.In that framework, 17 user expectations in four dimensions, given in Table 1, were examined.Each evaluation variable was represented in evaluation result by the degree of its importance to end users.The details of the framework and the results of the case study in two hospitals about expectation meeting ratio (EMR)s were given in the previous work (Gürsel , Zayim , Gülkesen , Arifoğlu , & Saka , 2014).The questionnaire, named as "Expectation Questionnaire", was used for data collection.In this questionnaire, there is an importance question to capture the importance of that variable to the user, and there is some number of questions to capture the expectation meeting, for each expectation variable.Users were supposed to express their importance values using five-point Likert scale (very important, important, average important, not so important, not important), and expectation values using five-point Likert scale (strongly agree, moderately agree, not sure, moderately disagree, strongly disagree) which are our linguistic variables in fuzzy logic operations.In the previous work, that questionnaire is applied to two different hospitals.The data gathered by Expectation Questionnaire contain a weight (importance degree) and expectations ratings for each variable, for a user.
In the previous work summarized above, in computation of the Expectation Meeting Ratio (EMR) of the systems, fuzzy logic methodologies are used.This fuzzy logic usage has aroused some questions: "What if we use classical computation method using crisp values?","Does the use of fuzzy logic make any difference and contribution?"In this study we seek answers to these questions.The aim of this study is investigating the difference and contribution of use of fuzzy logic in HIS evaluation.

Materials and Methods
In this study, some of the data captured in the previous study (Gürsel , Zayim , Gülkesen , Arifoğlu , & Saka , 2014) by the Expectation Questionnaire is used.The data of the one big hospital having 504 questionnaires are used to compute EMRs.In addition to the EMRs computed using fuzzy logic methods in the previous study (in the rest of the study it will be called as Fuzzy Computation), EMRs are recomputed using the same data with crisp values by classical computation method (in the rest of the study it will be called as Crisp Computation).Two sets are compared by the help of statistics.

Fuzzy Computation
In Fuzzy Computation, importance weights and expectation ratings and are fuzzified, by converting into fuzzy triangular numbers.These fuzzy numbers are used to compute the EMRs.Fuzzy triangular numbers assigned to the weight and rating Likert scales are given in Table 2.In the second part, EMRs are computed using crisp values, for importance weights, assignment is made ranging from 5 = very important to 1= not important, for expectation ratings, assignment is made 2 = strongly agree to -2 = strongly disagree (Crisp computation).The formulas used for both computations are the same and given below.
The final importance weight, Wi , of an expectation variable i can be given as where n is the number of users.
By aggregating the expectation rating answers given by the users, the final rating, Ri of the expectation variable i can be given as where n is the number of users.
In both computation methods, now we have 17 weights and 17 Ratings.The EMR is obtained by the weighted average formula where n is the number of variables, W is the weight and R is the final rating of the variable k .
In the Fuzzy computation, operations in all equations are fuzzy operations, so resulting EMR is also a fuzzy triangular number.It must be defuzzified to have meaning in the real world.For defuzzification, Best Non-fuzzy Performance method of Centre of Area (COA) defuzzifier [16] is used, to convert EMR into crisp number.
For both methods, now we have crisp values as EMRs.Finally by making a ratio between the possible lowest EMR and possible highest EMR of each method, the percentage of EMR is get.

Statistical Analysis
Statistical Package for Social Sciences 19.0 (SPSS, SPSS Inc, Chicago, Illinois, USA) is used for statistical analysis.
The internal consistencies of the answers to the "Expectation Questionnaire" are measured by Cronbach´s Alpha coefficient.Cronbach´s Alpha greater than 0.70 is considered reliable.To examine the difference between the results of two methods, chi-square test is used; p < 0.05 is considered as statistically significant.

Results
The Cronbach's alpha value for the importance weights is 0.871, for the expectation ratings is 0.966, showing that the answers to the questions are internally consistent.
The results of EMRs of both computation and the test results of their comparison are given in Table 3.In the Fuzzy computation, the variables with lowest EMRs are Ease of Use (with 27.89%) and Bringing Positive Change (with 28.82%).Variables with the highest EMRs appear to be Availability (with 62.96%) and Security (with 58.07%).In the Crisp computation, variables with the lowest EMRs are Speed (with 22.26%) and Bringing Positive Change (with 30.07%) whereas variables with highest EMRs are Security (with 61.51%) and Privacy (with 56.95%).General EMR values are 40.36%for Fuzzy computation and 42.27% for Crisp computation.
In Table 3, the variables having statistically significant difference are highlighted in the p column as bold.In 8 of the 17 variables, Ease of use, User Support, Speed, Privacy, Availability, Interoperability, Reporting Facilities, Research Facilities, test results shows statistically significant difference between two computation methods ( p < 0.05).Ease of Use, Privacy, Interoperability, Research Facilities expectations become lower in crisp computation when compared fuzzy computation, whereas User support, Speed , Availability, Reporting Facilities expectations become higher in crisp computation when compared fuzzy computation.
In each expectation dimension, there is at least one expectation having statistically significant difference.

Discussion
Fuzzy logic applications/methodologies are very suitable for HIS applications and there are many examples, but not employed in the evaluation frameworks.In this study the first fuzzy logic use in HIS evaluation is under examination.The results of tests show that there is a difference between classical crisp computation and fuzzy computation.
In Table 3, it is seen that there is an obvious difference between EMRs calculated by Fuzzy and Crisp computation.Differences in EMRs of Ease of use, User Support, Speed, Availability, Interoperability, Reporting Facilities, and Research Facilities are statistically significant.The lowest and highest EMRs are also different in two computations.In the Fuzzy computation, the lowest variable is Ease of use with 27.89 % whereas in the Crisp computation Speed becomes lowest with 22.26%.In the Fuzzy computation, the highest variable is Availability with 62.96 % whereas in the Crisp computation Security becomes highest with 61.51%.
The direction of the difference is not stable.It is seen that the four of the eight having statistically significant difference is higher while the other four are lower.There is no explanation for this result in the findings of the study.This issue may be further study point.
If the researchers prefer using crisp values than fuzzy numbers, the results may be misleading as in the given example.Use of fuzzy logic methodologies gives us elasticity for evaluation.In rating based evaluations it is easier to express the ratings linguistically rather than using numbers.In that case problem of ambiguity could emerge as in the rating "Not sure", which has negative meaning as "a little Disagree", neutral meaning as "No idea" and also positive meaning as "a little Agree".This detail will be missed if we use crisp rating values.Fuzzy logic gives us the opportunity to take these blurred boundaries into consideration (Lee, San, & Hsu, 2011).With this approach, the framework possibly became more realistic by covering uncertainty of the weights and ratings.
It is mentioned about the complexity of the Medical Domain in the introduction part.Taking into consideration of the complexity, in the evaluation of healthcare systems, we cannot lose precision by using crisp values and strict lines.We think that our method is more elastic, precise and detailed by the help of fuzzy logic methodologies.

Conclusion
In this study, contribution and difference of use of fuzzy logic in HIS evaluation is examined.
Use of fuzzy logic for evaluation of Likert type HIS user satisfaction questionnaire shows statistically significant difference from conventional crisp methodology.We believe that fuzzy logic helps to obtain a more realistic evaluation by taking blurred boundaries into consideration.

Table 2 -
Triangular fuzzy numbers assigned to Likert scales

Table 3 -
Crisp numbers assigned to Likert scales