Efficiency Measurement of Project Management Software Usage at State Social Insurance Agency

Abstract One of the activities for improvement of project management (PM) quality is to introduce or change PM software to a more suitable one for an appropriate project and team. After implementation of the new PM software, it is useful to understand real improvement and effectiveness of the PM software usage. The objective of the present research is to identify characteristics and methods for effectiveness measurement of the PM software usage and demonstrate its application for the case study to measure project team efficiency after the PM software implementation and assess impact of the PM software usage on the project performance. Mann-Whitney test and Spearman correlation coefficient have been used to analyse relevance between the PM software usage characteristics and project performance indicators with the purpose to determine the PM software usage impact on the project performance and PM quality. The case study has been performed at the State Social Insurance Agency. In order to increase PM quality, Redmine (free of charge PM software) has been implemented.


I. INTRODUCTION
With an increase of information technology (IT) usage, inside a company IT implementation in daily processes and fast and effective decision making become of great importance [1], [3].One of the important keys in successful achievement of this target is well-organised project management (PM) of IT implementation projects.It could be reached by using project management information systems (PMISs) during project planning, execution and control.PMIS affords tools to a project manager for team managing and provides a company with new information and knowledge to which managers should pay special attention [3].
PMIS provides a company with several advantages (quality improvement, control, scheduling, evaluation, staff management, and risk management), but not always the usage of PMIS transforms in positive gain [4], [12], [6] Unsuccessfully chosen PMIS could not give the desired improvement in PM, project progress and team performance that may be related to difficulties of PMIS usage due to a lack of knowledge or PMIS does not fit the project needs.Not always companies measure project characteristics and evaluate real effectiveness improvement by using PMIS after implementation of PMIS and using it for some time.As efficiency is an undefined and unknown indicator, it is hard to determine if PM becomes more qualitative and the project team -more effective.To make sure about project performance, it is necessary to find out its metrics, make measurements and determine impact on project performance.
Thus, the objective of the present research is to identify characteristics and methods for effectiveness measurement of the PMIS usage and demonstrate its application for the case study to measure project team efficiency after the PM software implementation and assess impact of the PMIS usage on the project performance.
For the industry case study, the authors have used the data warehouse team of the State Social Insurance Agency (SSIA).With the purpose to increase PM efficiency of the SSIA data warehouse team, Redmine [6] was configured according to project and team requirements and implemented one year before evaluation.
The rest of the paper is structured as follows: Section 2 describes the related research on efficiency measurement of PMIS usage identified using a systematic literature review; Section 3 presents a case study description and procedure of measurement collection and analysis; results of the case study are given and discussed in Section 4. Conclusion and limitation of the research are discussed at the end of the paper.

II. RELATED RESEARCH
To identify characteristics and methods that can be used for the effectiveness measurement of the PMIS usage, we have investigated the existing studies in this direction.To make a literature analysis enough effective, the systematic literature review [8] principles have been used.The process of literature review has been described in Section II.A, and the results have been discussed in Section II.B.

A. Process
Following the systematic literature review [8] principles, the following steps have been performed.
Step 1. Research questions have been determined to which answers we would like to find:  How could performance of PMIS be valued and what metrics have been measured and analysed? What is the PMIS impact on a project? What information could a company obtain after PMIS implementation? What is the experience of PMIS implementation in data warehouse? What are the prerequisites for correct measurement of PMIS performance?Step 2. Keywords for related article search have been defined: "Project management tools efficiency metrics" OR "Project management tools efficiency factor" OR "Measuring project management tools efficiency" OR Information Technology and Management Science _______________________________________________________________________________________________ 2016/19 66 "Increasing efficiency of project management" OR "Implementing project management tool" OR "Successful project management tool implementing" OR "Advantage of project management tools" OR "Project management tools impact" OR "Project management information systems efficiency factor" OR "Project management information systems effectiveness factor" OR "Measuring project management information systems efficiency" OR "Implementing project management information systems" OR "Project management tools" OR "Project management information systems" OR "Project management software" OR "Project management system" OR "Project management software deployment in data warehouse".
Step 3. Article search in scientific databases have been performed.The four databases (IEEE Xplore, Science Direct, Web of Science, Springer, Scopus) have been used, and articles containing the mentioned keywords in headings, abstracts or metadata have been selected.Results of article search are summarised in Table I.Articles have been classified by year to find out the popularity of research topic.Article distribution by the period of publication in Scopus is shown in Fig. 1.It shows that the topic of PMIS implementation and efficiency was popular after 2007, when the popularity of the PM software increased [10].Distribution of articles by science fields in Scopus is illustrated in Fig. 2. Articles with the above-mentioned keywords have frequently been met in computer science, engineering science and business.
Step 4. Article filtering according to inclusion requirements:  Article contains information about at least one PM efficiency measurement method, process and results;  Described tools and methods must be related to information technology management;  All content must be available in full text.After article filtering, about 1 % of articles have remained.The number of articles has noticeably decreased after applying the third inclusion requirement to 100 articles.

B. Results
Researches that have been related to IT/IS impact on company performed measurement and analysis to determine influence on productivity, process optimization, innovation implementation.Some studies describe only IT influence on a company, for example, supply channel or country's economic development [10], [11].In [12], the relation between users' satisfaction with PMIS and PMIS itself and its quality is described, and it has been detected that PMIS quality significantly influences users' satisfaction that is the primary issue of PMIS effective use.In one of the studies, it has been determined that PMIS provides the project manager with such advantages as project planning, monitoring, controlling, and decision making [13].Positive impact on PMIS performance is exerted by users who are well informed about data managed in PMIS [14].
Methods like Mann-Whitney U-test and Spearman correlation coefficient have been used for the purpose of analysis [11].Mann-Whitney metric is a non-parametric test of differences in means, and it is used since the distribution of the population is unknown and the sizes of three subsamples are small.Spearman correlation coefficient is a nonparametric measure of rank correlation.This metric is used to evaluate dependence of two indicators if normal distribution has not been proven and one or both indicators have been evaluated using range scale [15].To make calculations, the working table is created where the first and the second columns consist of related independent and fruitful indicator values -x i and y i .The third and the fourth columns contain values of ranges -g x.i and g y.i .The fifth column contains related match range differences -d i = g x.i -g y.i .The sixth  (1) Among the found articles, there is no article that describes PMIS implementation impact on data warehouse PM.Moreover, the found articles are not mostly based on real data.Data warehouse project is specific as one kind of its developing products is universes and reports based on them.

III. CASE STUDY DESIGN
The case study has been performed in SSIA data warehouse projects where free of charge PM software Redmine was implemented one year before evaluation (detailed description of case has been given in Section 3.A).During the evaluation procedure, data collection and analysis activities have been performed (Section 3.B) to measure project team performance changes and assess impact of the PM software usage on project performance after Redmine implementation.

A. Case Description
Redmine was implemented to prevent SSIA data warehouse team from the following problems:  Project manager did not know what exact activities were made by team members in each case;  There was no common overview of the completed and scheduled tasks;  It was impossible to manage the team effectively enough because of lack of time management tools, etc. Redmine is used for project, incident and risk management.This tool is open source and widely customisable.It was written by Jean-Philippe Lang in 2006 [6].Redmine is a web based application whose server could be placed both on Windows and Linux, Unix, CentOS, etc. Redmine stores its data in one of the following databases -MySQL, PostgreSQL or SQLite.It uses Ruby on Rails framework and one of the following web servers -Apache, Apache, Ngnix, and Webrick.
As shown in Fig. 3, Redmine consists of four modules where each is responsible for specific information part.Project documentation module includes document adding, deletion and editing.This module also includes Wiki pages where tutorials and tips are stored.Management information module includes task, its progress management and time logging.PM module includes internal repository and event management.Communication module is responsible for any kind of contacts between team members.This module is well developed providing the members involved with email messaging for task status or field value change.

B. Data Collection and Analysis Procedure
The study consists of two flows shown in Fig. 4 where the first provides steps for data collection and the secondanalysis.
The data collection flow describes all data preparation steps and compilation formulas (Fig. 4).Three global variables have been chosen for data collection that consists of few metrics.Global variables describe PM and project indicators, which are shown in Table II.The metrics used in this case study has been chosen in accordance with similar studies about the PM software / PMIS impact on project [16], [6].The data collection includes the following steps (Fig. 4).
Step 1: Project data collection and compilation.The number of involved team members and duration of project have been used for evaluation of the project metrics.Product is a unit processed inside a project and described by product complexity that is a sum of product objects, formulas, reports divided by duration of a project.Step 2: Project performance data collection and compilation.Cost performance index (CPI) has been used as the project performance metrics that corresponds to the ratio of the budget cost of work performed to the actual cost of work performed.
Step 3: Project classification.In the data warehouse project, threshold values of CPI (a = 0.9 and b = 0.8) have been defined by the project manager that divides projects into groups by CPI according to Table III.Step 4: Statistical data collection and compilation of the PM software by modules.PC Pandora was used for frequency and duration of use measurement of each Redmine module, which could log all computer usage statistics and provide HTML type reports.Program was configured to log hits on websites and spent time inside each of it.As each module has a definite site so that it was easy to determine and measure the intensity of use and spent time.
The PM software metrics have been evaluated by division of frequency of use of modules and duration of a project in days as well as by division of duration of use of module and duration of a project.
The second flow (Fig. 4) supports the analysis of collected data.The analysis is based on methodology, which assumes positive relation between an ideal profile and performance [17].In order to develop an ideal profile, it is necessary to use system metric mean score values and divide them into appropriate performance group.Such an empirically derived profile is close to the concept of strategic benchmarking, rather straightforward and intuitively appealing [17].To detect an ideal profile, mean duration of tool usage has been determined.
The analysis includes the following four steps (Fig. 4).
Step 1: Analysis of usage frequency of the PM software modules comparing metrics by performance groups (weak CPI, study CPI, good CPI) with Mann-Whitney test.This test has been chosen because distribution of the population is unknown and the sizes of the three groups are small.Mean usage frequency of the PM software modules for each performance group has been determined by dividing connection count of module in each project by project duration.The performance groups have been compared using Mann-Whitney test for each module to determine difference significance of value distribution of mean usage frequency.
Step 2: Analysis of usage duration of the PM software modules comparing metrics by performance groups (weak CPI, study CPI, good CPI) with Mann-Whitney test.Mean usage duration of the PM software for each performance group has been determined by dividing duration of connection for each project by its duration.Then values of performance groups have been compared using Mann-Whitney test to determine difference significance of value distribution of mean usage duration.
Step 3: Analysis of mean scores of usage frequency of PM software and product complexity correlation by performance groups (weak CPI, study CPI, good CPI) using Spearman test.First, mean values of previously determined mean usage frequency of the PM software modules have been calculated.Then these mean value have been compared with product complexity by the performance groups using Spearman test.
Step 4: Analysis of usage frequency of the PM software modules and CPI correlation by performance groups (weak CPI, study CPI, good CPI) using Spearman test.The previously determined mean values of usage frequency of the PM software module have been compared with CPI values using Spearman test.The positive correlation values indicate that the more tools are used the better project performance is.

IV. CASE STUDY RESULTS
This section describes results of the case data collection and analysis according to the previously defined procedure (Section II.B).
The case study metrics have been collected from 15 different projects.Each project used the same Redmine modules: Project documentation, Management information, Project management, Communication.Statistics were analysed by MS Excel.Table IV demonstrates mean score and standard deviation of main project metrics.
Table V shows mean scores of usage frequency of Redmine modules by project groups (WCPI, SCPI, GCPI) as well as significance levels (p) according to Mann-Whitney test (nonparametric test of differences in means).Grouping of projects has been made according to CPI threshold values: WCPI include 4 projects, SCPI -7 projects, GCPI -4 projects.Results show that comparing WCPI and GCPI (ideal profile scores) for 2 of 4 modules a significant difference has been observed.Comparing WCPI and SCPI, significant changes have been observed for all modules.Mean metric scores of SCPI group are greater than ideal profile (GCPI), which means that the stated level of Redmine use does not allow reaching ideal profile results due to module usage specifics.Insignificant difference has also been observed between SCPI and GCPI.
Mean scores of usage duration of Redmine module in each project group (WCPI, SCPI, GCPI), as well as significance level values (p) that have been calculated comparing groups using Mann-Whitney test are shown in Table VI.Correlation between usage frequency of Redmine module and project performance is shown in Table VIII.Correlation coefficients demonstrate that often the usage of Redmine has been observed for project with the greatest performance.The greatest correlation has been observed for a communication module that also has the greatest impact on project performance.

V. CONCLUSION
This paper demonstrates the case study of impact analysis of Redmine usage to project performance by using Mann-Whitney test and Spearman correlation coefficient.Fifteen projects of data warehouse team have been observed, data about Redmine use (usage frequency and usage duration) have been collected, project metrics and product complexity analysed.The main conclusions of Redmine usage efficiency evaluation are as follows:  The longer Redmine is used, the better CPI results are, i.e., the studied tool has a positive effect on the project success. Greater correlation was between usage frequency of communication module and project performance, which means that this module has the greatest impact on project performance. Increasing project performance also increases usage frequency of Redmine.Similar results have also been observed in other studies [18], [4].
One of the limitations of this case study is that duration of use of a website is not exact enough because a user can open a website and leave it for a long time and it does not mean that the page has been really used.Case study has only 15 projects and it is not enough for general conclusions about PMIS usage effectiveness improvement.The evaluated projects can be used as a basis for other studies.
Future studies will be related to adding additional metrics, analysing another performance indicator instead of CPI.

Fig. 1 .
Fig. 1.Research article count by year in Scopus database.

Table VII .
We have observed that by increasing project complexity the usage frequency of Redmine also increases.The greatest coefficient to GCPI proves this statement.