Mapping open knowledge institutions: an exploratory analysis of Australian universities

Descriptive data analysis

Our analysis starts with some individual descriptive statistics, and seeks insight into patterns emerging from each indicator and comparisons thereof. A number of summary statistics are recorded in Table D1 of Appendix D for reference. Here we focus on several statistical descriptions that are more comparative across the different measurements. Figure 1 below presents each indicator’s level of skewness, level of kurtosis, and its p-value resulting from the Shapiro–Wilk normality test. Many of the indicators exhibit substantial skewness (deviation from 0), and are leptokurtic (kurtosis value greater than the normal distribution, i.e., 3). These characteristics are consistent with the low p-values obtained for the corresponding Shapiro-Wilk test, indicating that many of the indicators are highly unlikely to be normal in distribution.

The proportion of “Indigenous staff” displays the highest levels in both skewness and kurtosis. It is positively skewed due to two universities having 16.13% and 4.4% Indigenous staff, respectively, compared to all other universities sitting in the 0% to 2.4% range. Other indicators with highest levels of skewness and kurtosis include “Gold OA”, “Collaboration beyond Australian universities ”, “Women above rank of senior lecturer”, and “Women in non-academic roles”. For each of these indicators, the data contains extreme observations (these are not outliers due to error and cannot be simply dropped). These extreme data points may have significant influences on the measures presented in Fig. 1.

For an alternative view of each indicator’s distribution, we construct their respective histograms. We use the Freedman-Diaconis rule for binwidths (hence, the corresponding number of bins) which is less sensitive to extreme observation than the standard Sturges’ formula for number of bins. This is applied to all indicators other than “Policies on library access”, “Policies on OA” and “Policies on diversity”, where the values of the original observations (i.e., scores) are used as bins. These histograms are displayed in Fig. 2.

As mentioned, many of the indicators exhibit extreme observations. However, even with these extreme observations ignored (see Figure D1 of Appendix D for corresponding boxplots of normalized observations), many indicators are still characterised by substantial skewness (i.e., asymmetry) and high levels of kurtosis (i.e., fat tails). The above results imply the use of robust statistical methods that cater for non-normality and extreme observations is necessary for analysing this set of data.

To better understand each indicator, it is worth discussing some of these extreme observations. For example, the single extremely large observation for “Gold OA” corresponds to a significantly smaller institution in output size where 2 out of 3 publications were made OA via the publisher. The same university is responsible for the largest observations in “Total collaboration”, “Collaboration with Australian universities”, “Indigenous staff”, “Women above rank of senior lecturer”, “Women at rank of senior lecturer”, and the lowest values for “Collaboration beyond Australian universities”, “Industry collaboration”. Similarly, three of the smallest universities have 0% for “Bronze OA” and two of the smallest universities also contribute to the two lowest values in “Total events”. However, some extreme observations also result from other larger universities. For example, a medium to large sized university resulted in the largest value for “Annual report diversity score”.

The existence of extreme observations, together with highly diverse measurement units across the indicators, can easily distort standard measures of spread, such as variance and coefficient of variation. Hence, we supplement our analysis by calculating the quartile coefficient of dispersion (QCD) for each indicator. QCD is a unitless measure of dispersion that is more robust against extreme observations. This allows us to make an overall comparison of the relative dispersion across the indicators, and also study the level of disparity within each indicator. The results are displayed in Fig. 3, together with confidence interval (Bonett’s confidence interval, as appropriate for the sample sizes here; Bonett, 2006; Altunkaynak & Gamgam, 2019) for the QCD value in each case.

Immediately we observe that “Total revenue” and “Indigenous staff” have the highest values for QCD, an indication of their high inequalities across the universities. The large inequality of “Total revenue” is highly correlated with output size (i.e., 0.97 in Spearman’s coefficient of rank correlation). This is immediately followed by the high level of disparity in the proportion of Indigenous staff (even with the effects of the extreme observations minimised by QCD). “Annual report communication score” also has a high QCD value, signalling the large differences across university annual reports in terms of the proportions of phrases or keywords related to communication. The “Walk score” indicator has the next highest QCD, which is likely attributed to the large differences in physical accessibility due to locations of the universities.

The proportion of collaborative publications (“Total collaboration”) and the proportion of non-academic women staff (“Women in non-academic roles”) have the lowest QCD value (ignoring the trivial case of “Policy on diversity”). This is a result of high concentration of values around the central part of their respective distributions (around 74% and 66% respectively). Overall, there is a high degree of differences in disparity across the indicators.

Correlation analysis

In this section, we explore the potential relationships between the OKI indicators. We calculate the (zero-order) Spearman’s rank correlation coefficient between all pairs of indicators (missing values are left out and only pairwise-complete observations are included for each pairwise calculation). The overall result is summarised into a network plot presented in Fig. 4, using multidimensional clustering. In the Figure, indicators that are more highly correlated appear closer together, and are joined by edges with darker shades. Blue edges indicate positive correlations, while red edges represent the negative correlations.

In Montgomery et al. (2021), an evaluation framework for OKIs is proposed by characterising indicators into three platforms: diversity, communication and coordination. This is combined with a theory of change that evolves through the three stages of aspiration, action and outcomes. It also noted that the indicators may become increasingly more difficult to characterise into the three platforms as we move through the three stages of change. Figure 4 represents a practical example of the above. In the bottom right, we see a cluster of indicators mainly related to diversity (i.e., gender and Indigenous proportions). The bottom left section of the Figure encompasses a group of indicators related to communication, such as OA and collaboration. And, perhaps more dispersed, is a set of indicators for coordination (e.g., policies) gathered at the top of Fig. 4. Note that “Total revenue” is not included in Fig. 4, as it is not directly considered as an OKI indicator. We will take a closer look at its correlations with the OKI indicators later. It should be noted that the proximities of the points in Fig. 4 are determined by multiple clustering. This means where a point lies is relative to its magnitudes of correlations with all other points. Hence, direct comparisons between correlations of different pairs of indicators must be made with caution.

A few indicators appear less clearly defined in terms of which platforms they relate to, such as “Total events” and “Bronze OA”. This does not necessarily imply that they are not correlated with other OKI indicators. Rather, they may be similarly correlated (in magnitude) to indicators from multiple platforms making them less distinctive for classification. For example, “Total events” has similar magnitudes of correlation with both the OA indicators and the diversity indicators (see Fig. 5 for examples).

Another interesting observation is that “Output diversity” is closely located to many of the OA indicators and the collaboration indicators. These correlations are in the negative direction throughout. Two examples are given in Fig. 6. We should note here that most of the bibliographic data systems (including ours) capture more accurate information on journal articles than other output types. Also, given most universities have journal articles as the primary output format, the “Output diversity” indicator is mainly influenced by the inclusion of other output types, such as book chapters, conference proceedings, datasets, etc. The other output formats are also more likely to be recorded as non-OA¹ (noting that our data workflow only tracks objects assigned with Crossref DOIs and is dependent on whether Unpaywall is able to accurately track their accessibility options) and are affiliated to smaller numbers of authors. Hence, higher diversity in output types is associated with lower OA levels.

The indicators “Collaboration with Australian universities” and “Walk score” appear out of place at the first glance. They seem more highly correlated to the gender and Indigenous indicators, and less correlated to other indicators relating to collaboration and access. However, a deeper exploration reveals some interesting relations. The “Walk score” is negatively correlated with the diversity indicators, while “Collaboration with Australian universities” is correlated positively with the same indicators. Examples of these for “Women in academic roles” are given in Fig. 7. Note that “Walk score” can also be seen (in general) as a benchmark for the locations of universities. This implies universities in more regional areas (which also tend to have lower total revenue and are smaller in size) have proportionally more women and Indigenous staff. These universities also produce higher proportions of outputs co-authored with others in the list of 43 universities (hence a negative rank correlation between “Walk score” and “Collaboration with Australian universities”). In contrast, “Walk score” is positively correlated with the other collaboration indicators, albeit at relatively smaller magnitudes.

We also note the generally low correlation levels between OA indicators and the gender indicators. In comparison, Olejniczak & Wilson (2020) found that (for a sample of US universities) there is a slight bias towards male authors in terms of OA publications, and this bias increases if job security and level of resources are taken into account. Our data present little to no evidence for this bias in the Australian context, although further study is required.

Pairs of indicators with the highest positive values of rank correlation are “Total OA” against “Green OA” (0.89), and “Women at rank of lecturer” against “Women in academic roles” (0.86). In contrast, pairs with the highest negative correlation coefficients are “Gold OA” against “Output diversity” (−0.59), “Output diversity” against “Collaboration beyond Australian universities” (−0.57), and “Indigenous staff” against “Walk score” (−0.58). We have already had discussions related to pairs with highest negative correlations earlier. So we will now focus on pairs with the highest positive correlations. Figure 8 displays the scatterplots of the two pairs with the highest positive correlations.

Almost all universities (42 out of 43) in our study have a higher proportion of “Green OA” (i.e., repository-mediated OA) than “Gold OA” (i.e., publisher-mediated OA), which is not unexpected as most gold OA publications are available to be archived by repositories (albeit sometimes in the form of earlier manuscript versions). For example, Robinson-Garcia, Costas & van Leeuwen (2020) showed that the scale of which “Gold OA” outputs are also archived by repositories to be as high as 81%. Green OA is also seen as the more cost-effective route compared to gold OA due to the high cost of article processing charges (APCs). In contrast, 34 universities have higher values in “Gold OA” than in “Green only OA”. Australian universities also have a low uptake on “Bronze OA”. The strong focus on “Green OA”, relative to other OA types, explains the high correlation between “Total OA” and “Green OA” (note for example the rank correlation coefficient is 0.55 between “Total OA” and “Gold OA” and is 0.89 between “Total OA” and “Green OA”).

The strong correlation between “Women at rank of lecturer” and “Women in academic roles” requires a deeper exploration. Figure 9 provides an overview of the proportion of women at various levels of academic positions within all 43 universities. Evidence shows there are higher proportions of women staff at lower academic positions consistently across all institutions. For example, 40 out of the 43 universities have a higher proportion of women in positions below lecturer level than proportion of women above senior lecturer level. There is also a higher proportion of women in non-academic roles, where 42 out of the 43 universities have a higher value for “Women in non-academic roles” than for “Women in academic roles”. It is also higher than all other gender proportions for almost all universities (as depicted in Fig. 9). Note that we also observed earlier (Fig. 3) that “Women in non-academic roles” has one of the lowest QCD values among all indicators. Together with a much larger number of women employed at the lecturer level in academic roles, the above factors contribute to the high correlation observed between “Women at rank of lecturer” and “Women in academic roles”.

Lastly, we look at the potential influences of the university’s revenue on the OKI indicators. Figure 10 displays the rank correlation coefficient values between “Total revenue” and each of the other indicators. The three-way correlation between “Total revenue”, “Walk score” and “Indigenous staff” is consistent with our comment earlier regarding the sizes and locations of universities. That is, smaller, regional, and less wealthy universities are correlated with higher proportions of Indigenous staff. The negative correlation between “Total revenue” and all the gender indicators also displays a similar pattern. It is also worth noting that almost all universities in our data have existing policies on employment equity and diversity (as recorded by the “Policy on diversity” indicator). Evidently, some differences in outcomes exist across universities, but the levels of actioning (upon existing policies) remain difficult to quantify and require further exploration in the future.

The correlation of “Total revenue” versus “Collaboration with Australian universities” and “Collaboration beyond Australian universities”, respectively, are high but in opposite directions. Consistent with our earlier discussion, universities with lower total revenue appear to have more proportions of output co-authored with other universities in our list of 43 Australian universities. In contrast, the more wealthy universities seem to have higher collaboration proportions outside these 43 universities, including international universities and research organisations. The size of “Total revenue” seems to have little correlation with “Industry collaboration”, though this indicator is derived from an external source directly and should be interpreted with caution.

The generally low correlations between “Total revenue” and the OA indicators (except for “Bronze OA”) is an interesting outcome. They are a potential indication that higher revenues at Australian universities do not necessarily translate to higher proportions of OA outputs (which one might expect in relation to the provision of OA publishing funds, the cost of APCs and costs associated to managing an institutional repository). The findings do however conform to the fact that only 3 universities in our data have signalled a provision of funding for OA publishing. This, and the fact that all but one university have an OA repository, may explain the slight increase in correlation against “Green only OA”. We should also note that the OA indicators seem to have only low to moderate correlations with “Policies on OA”. The moderate correlation between “Bronze OA” and “Total revenue” poses an interesting case, where we find more wealthy universities to have higher proportions of Bronze OA publications. However, these proportions remain generally low for all universities in the study.

Principal component analysis

In this section, we apply PCA on the OKI indicators. This is aimed at providing insight into how information is attributed across the different indicators, and how these indicators relate to a few principal components (PCs). Due to the existence of extreme observations, missing data and vastly different measurement scales, with no existing knowledge of a plausible re-scaling method, we propose a two stage process for PCA.

Firstly, being constrained by the size of data, we propose imputing the missing values using an iterative PCA algorithm (implemented by using the R package missMDA, with the function estim_ncpPCA used to estimate the number of PCs to be used for imputation and the iterative PCA process excuted by the imputePCA function). This procedure uses the mean of each indicator as initial values for the missing data. Subsequently, a standard PCA is applied and a selected number of PCs are used to re-estimate the missing values. The process is repeated iteratively until the imputed values converge.

Once the imputed data is obtained, we proceed with robust PCA procedures that cater for extreme observations. Common approaches for this purpose include the use of robust covariance (or correlation) matrices and projection pursuit. An example of the former is the use of the Spearman’s rank correlation matrix (equivalent to applying standard PCA to ranks within each indicator) in the classical PCA procedure. For the latter, one can use the ROBPCA procedure which combines robust covariance matrix estimation with projection pursuit (Hubert, Rousseeuw & Branden, 2005). Given that the results are very similar, we report only according to the first method in this section. Supplementary results and results for the second PCA method are provided in Appendix E.

Figure 11 shows the percentages of variance explained by each of the PCs and the corresponding eigenvalues derived from the Spearman-PCA approach. For the left diagram, the red line records the cumulative percentages of variance. While the first two PCs display the highest proportion of variance explained, their combined coverage is merely above 40%. It is observed that we need at least 8 PCs to attain a variance coverage of approximately 80% for the Spearman PCA. Analogous results are observed using the Kaiser criterion (red line in the diagram on the right) on eigenvalues to determine the number of PCs to retain.

The low proportions of variance explained by the individual PCs and the high number of PCs needed to attain a significant coverage of the cumulative variance indicate that the set of OKI indicators provides diverse information where the overall variance is spread across multiple directions. However, keeping a high number of PCs also makes the interpretations of these PCs more difficult, as the later PCs show less distinctive groupings of loadings by the original indicators (see Tables E1 for loadings on the first 8). A common approach is to rotate the PCs according to a select first few PCs based on informed judgement. In this case, the first two PCs display resemblance of the OKI evaluation framework described earlier; with one of the first two PCs correlated largely to diversity indicators while the other with the communication indicators. To a lesser extent, there may be a third PC that relates to coordination. As a graphical illustration of this, the correlation circle plot against the first two PCs is provided in Fig. 12.

In the correlation circle, all 26 OKI indicators are projected onto the first two PCs. Angles between arrowed lines represent correlations between indicators in this plane (with 90 degrees indicating zero correlation and 180 degrees indicating perfect negative correlation). Lengths of the arrowed lines are indicative of how well they are represented in this two-dimensional space (or their levels of contribution to these PCs). In Fig. 12, many of the diversity indicators are pointing to the right along the first PC, while many of the communication indicators are pointing in the direction of the second PC. Many of the correlations represented in this correlation circle are also consistent with observations made in Fig. 4 earlier. Most of the coordination indicators have shorter arrowed lines, indicating they are not well-represented in this space, and are likely to be more representative by other PCs.

The observations made above lead to a focus on the first three PCs (a third PC included to try capture variances in the coordination indicators). Table E3 in Appendix E lists the loadings by each of the OKI indicators after rotation (performed using the varimax function in R) against the first three PCs. For a simplistic overview that aligns with the OKI evaluation framework, we summarise the results into what proportions of each PC’s variance (calculated by sums of squares of standardised loadings of the selected indicators) are loaded by each of the three groups of indicators (diversity indicators: “Indigenous staff”, all gender indicators, “Collaboration with Australian universities” and “Walk score”; communication indicators: “Total OA”, “Gold OA”, “Green OA”, “Green only OA”, “Output diversity”, “Total collaboration” and “Collab beyond Australian universities”; coordination indicators: “Website score” and policy and annual report indicators. Others: “Total events”, “Bronze OA”, “Industry collaboration”). The groupings are decided depending on observations made in Fig. 4, and in conjunction to the Spearman’s rank correlation matrix and PCA loadings. Three of the indicators, “Total events”, “Bronze OA” and “Industry collaboration”, with non-distinctive grouping, are listed separately. These are presented in Table 2.

The results in Table 2 re-affirms our discussion earlier regarding which groups of variables provide the most loadings on the first two PCs. In addition, we observe the significant level of loading of the coordination indicators on a third PC. The indicator “Total events” loads on the coordination PC. On the other hand, “Industry collaboration” and “Bronze OA” seem to have little influence on the first 3 PCs.

Cluster analysis

In this section we are interested in exploring whether there are specific groupings of universities that can be defined by the OKI indicators. This is important in performing likewise comparisons and for identifying different paths of OKIs. As an immediate follow up from the PCA analyses, we are able to construct individual component plots of universities mapped onto any pair of PCs. In these plots, universities having similar profiles (or scores) in terms of the selected PCs will be displayed closer together. Figure 13 displays the individual component plot for the first two PCs from the Spearman PCA, with universities colour-coded by state or territory (as per main campus location), i.e., Australian Capital Territory (ACT), Multi-state (AU), New South Wales (NSW), Northern Territory (NT), Queensland (QLD), South Australia (SA), Tasmania (TAS), Victoria (VIC) and Western Australia (WA). No immediate pattern arises in terms of universities from a common state or territory as each group seems to be randomly scattered.

Alternatively, we assign a colour to each university by their affiliation to existing Australian university networks (i.e., Australian Technology Network (ATN), Group of Eight (Go8), Innovative Research Universities (IRU), Regional Universities Network (RUN), and with universities unaffiliated to any network grouping labelled as “None”) in the same plot. This is presented in Fig. 14. The standout group is the Go8 where all 8 member universities lie towards the top-left of the plot. This is an indication that these universities are quite similar in terms of their performance in the first PC (diversity) and the second PC (communication). Their overall performance leans toward the top half in communication, but tends toward the opposite direction for diversity.

We next consider cluster analysis of universities using the full set of OKI indicators and the respective ranks. Columns are standardized (through subtracting by the column mean and dividing by the column’s mean absolute deviation) to cater for the differing units of measurement. Hierarchical clustering is implemented by using the Manhattan distance (only pairwise-complete observations are used) to construct the dissimilarity matrix between universities, and the complete-linkage criterion is used to select similar clusters (implemented using a number of functions and packages in R: the daisy function from cluster is used to calculate the Manhattan distance matrix; hclust is used for the cluster analysis and results are subsequently converted to a dendrogram object for plotting using the dendextend package). These selections are made based on their robustness against extreme values.

Dendrograms of the clustering results are presented in Figs. 15 and 16. The former is derived using ranks as input, while the latter uses the original observations. In both figures, the university labels are colour-coded by university network affiliations as before. Corresponding figures colour-coded by states and territories are given in Appendix E. The heights of the vertical lines indicate the order in which the clusters were joined, and that order is determined by how similar two universities are in terms of their performances (or ranks) in the OKI indicators. Universities joined with lines at lower heights are considered more similar to each other.

As before, the university networks reveal more synchronised groups in comparison to locations. The most prevalent case is that of the Go8 universities. In both figures, these universities appear to be closely clustered. This is a potential indication of the synergies across universities in common affiliated networks.

A further interesting observation is related to those universities that largely remain in singular or very small clusters. These extreme cases are less obvious in Fig. 15 given the use of ranks removes the size effects. However, both figures consistently show that many of the last few universities to be added to clusters are small, private or specialist universities. Intuitively this makes sense given that such universities may have less resources, missions that deviate from traditional universities, and practices that need to be aligned to these.

Main findings and implications

In the previous section, we examined patterns and potential relationships across a number of OKI indicators for 43 Australian universities. We also explored ways in which information provided by these indicators can be summarised into a small number of orthogonal variables (PCs) in combination with the evaluation framework proposed by Montgomery et al. (2021). Universities were also clustered by using their corresponding data, ranks, and the corresponding PCs to reveal overall similarities across universities based on the OKI indicators. Based on these results, we reflect below on the research questions set out in the “Introduction” section.

First, we have found most of the OKI indicators to be clustered around the three platforms (i.e., communication, diversity and coordination) as described by Montgomery et al. (2021). This is signalled through both the correlation analysis and the PCA results. The network plot shows evidence of the three groupings through multiple clustering of the Spearman’s rank correlation coefficients. A few indicators (e.g., “Total events”, “Bronze OA” and “Industry collaboration”) seem less clearly positioned as to which platform they belong to. These are indicators with similar correlation levels against two or more of the three groups. However, if we consider these as signals of action or outcome (rather than aspiration), then their indistinctive positioning conforms to the theory of change within the OKI evaluation framework (Montgomery et al., 2021). For example, “Total events” is likely to be an outcome of mixtures between policy actioning (evaluation of “impact”) and communication (online accessibility to research), but also diversity (disciplinary practices), making its classification into the three platforms less trivial.

Through initial PCA results, we found that a higher number of PCs is needed to capture a reasonable proportion of the total variance. However, mappings of the OKI indicators onto the first few PCs show many diversity indicators aligning with the first PC and the communication indicators aligning with the second PC. Guided by this, the network plot and the OKI evaluation framework, we further rotated the PCs against the first three PCs. Examining the loadings onto the three rotated PCs reveals that diversity indicators make up most of the first rotated PC’s variance. Similarly, communication and coordination indicators make up most of the variances in the second and third rotated PCs respectively. These provide ways in which information underlying these indicators can be summarised, and demonstrate the use of the OKI evaluation framework as a potential tool for mapping performances of OKIs.

The second research question pertains to the understanding of relationships between the OKI indicators and their use in capturing the complexity of OKIs. Statistical analysis shows that many indicators are characterised by skewness and extreme observations. Hence, robust methods are needed to examine relationships across these indicators. The results also reveal large disparities across universities in terms of their performances in these indicators.

Investigation of correlations across these indicators finds interesting trends. Some trends are consistent with existing literature, while others indicate a shift of spectrum in the Australian context. For example, our data shows that Australian universities have relatively more of their research outputs made accessible via the “Green OA” route than the “Gold OA route”. This focus on “Green OA” is in agreement with trends on a more global scale for 2017 (Piwowar et al., 2018; Montgomery et al., 2021). This is likely a result of policy implementation and resource allocation in the Australian higher education sector. In comparison, there are regions for which “Gold OA” dominates the OA publication landscape, such as Latin America, based on pro-active policies and the SciELO (Scientific Electronic Library Online) cooperative OA publishing model (Huang et al., 2020b).

We also find increases in gender inequality moving up in academic ranks and between non-academic and academic positions. These are consistent with existing literature that shows progress for gender diversity to remain at more junior and non-academic positions within universities (Winslow & Davis, 2016; Baker, 2016). This occurs despite numerous diversity, equity and inclusion policies and action plans in operation at universities (Khan et al., 2019; Marini & Meschitti, 2018; Subbaye & Vithal, 2017; Winslow & Davis, 2016; Baker, 2016). While there has been progress on gender equity in the Australian context (Larkins, 2018), our findings suggest that the current level of diversity in academic positions is disparate across universities. In contrast, the proportion of “Women in non-academic roles” is quite consistent across universities (with the lowest QCD value out of all indicators). There also appear to be negative correlations between all diversity indicators (including “Indigenous”) and “Walk score” (which is a benchmark for location). These trends re-emphasise barriers to progress in hiring practices and achieving equity and parity (especially at traditionally high-ranking institutions). They indicate much work remains for advancing employment diversity in the Australian academic landscape.

Correlations of the OKI indicators against “Total revenue” also show several interesting patterns. Universities with higher “Total revenue” (also traditionally more prestigious) are correlated with higher proportions of “Collaboration beyond Australian universities”. The opposite is true for lower ranked universities who produce more proportions of “Collaboration with Australian universities”. This may be attributed to the fact that international collaboration enhances an institution’s reputation, impact, and ability to attract research and development investments, and research talents through both researchers and students (Australian Academy of Humanities, 2015; Glänzel, 2001; Glänzel & de Lange, 2002). These in turn influence the university’s position in international rankings. There is also a low correlation between “Total revenue” and “Gold OA”. In contrast, Siler et al. (2018) suggests gold OA publishing to be correlated to levels of funding and university ranks (although their finding is restricted to the field of Global Health research). This again conforms to Australia’s stronger focus on repository-mediated (Green) access to research outputs.

The above evidence suggests that analysis of these indicators in combination can support existing understandings of universities as OKIs and prompt new ways to interconnect these understandings. While each indicator is valuable in its own right, as a connected set of measurements, they can depict the complexity of OKIs. This offers novel insights into connections between these indicators and implications for universities’ progression as OKIs.

Access to these indicators provides alternative views on university performance. These multidimensional perspectives differ significantly from traditional university league tables that are more unidimensional. Indeed, our PCA results show that low levels of variance are explained by individual PCs before rotation and a high number of PCs is needed to explain a significant portion of the total variance. These imply that the information provided by these OKI indicators are high dimensional and complex. This is in contrast to findings related to popular university rankings where only one or two PCs are needed to capture most of the total variance in ranking measures (Dehon, McCathie & Verardi, 2010; Docampo, 2011; Johnes, 2018; Selten et al., 2020).

The complexity inherent in these indicators can be simplified through the use of the OKI evaluation framework. Together with cluster analysis, we are able to capture comparative performances of Australian universities on different groups of OKI indicators. There are both similarities and dissimilarities across universities, indicating their progress on a variety of paths as OKIs. Our results also suggest that classification of these universities as OKIs may be aligned with the Australian university network groups and levels of wealth and prestige. For example, universities in the Go8 are frontrunners in many of the communication indicators, but are relatively less successful in the diversity indicators. Smaller and less wealthy universities seem to illustrate higher performance in diversity but vary widely in their communication scores. In contrast, small, private and specialist universities portray performance patterns that are often unique compared to other universities.

Limitations and further research

While this study aims to be as comprehensive as possible in terms of both data collection and data analysis, a number of limitations need to be noted. Research outputs counted as part of this work are limited to those with existing DOIs that map with our Crossref data snapshot. Hence, a research output without a recorded Crossref DOI is left out. Affiliations between research outputs and universities come from three potential sources: Microsoft Academic, Web of Science, and Scopus. Each of these sources has their own limitations in accurately recording author affiliation information. Readers are referred to Huang et al. (2020a) for a report on such limitations. The bibliographic data sources are also dynamic (i.e., continuously changing), including potential backfilling. Our dataset focuses on the year 2017 with some exceptions. These are “Industry collaboration” and “Walk score”, where the former is reported by the CWTS Leiden Rankings for research outputs published over the period 2014 to 2017 and the latter represents scores shown at the time of access (2019) to the Walk Score website tool.

There exist missing values in the data that we have collected for the set of indicators. We have implemented and trialled various methods for the robust handling of these missing values. The results obtained are largely consistent across different methods. However, we cannot discard the possibility for the (unobserved) real data associated with these missing values to be vastly different and to potentially change the results significantly. Similarly, there are a number of extreme observations in the data, including those driven by sample sizes. While we have used robust methods to counter the potential size effects of these extremes, they nevertheless remain the largest or smallest values within the respective set of indicator observations.

We used extensive reviewing and manual work for the document analysis of university policies and statements for constructing the policy indicators. However, we cannot completely discard the possible subjectivities on our part of determining scores for these indicators. The annual report analysis is tested on two separate machines using different Python versions, and have yielded similar general results. However, we note that the process of transforming PDF documents to text files can depend on the operating system and versions of software used.

We have also subjectively chosen to use “Total revenue” as an indicator for university finance, size and prestige, but could have easily used other financial performance indicators (such as total expense, total grant income, fees and charges, employee benefits, etc.) from the Department of Education, Skills and Employment’s report. However, most of these indicators are highly correlated, e.g., the correlation between “Total revenue” and total expenses is more than 0.99, meaning they will likely lead to very similar results as reported in this article. Nonetheless, further explorations may be needed for less correlated financial indicators.

Lastly, our analysis is focused on the small sample of Australian universities for a specific year. Further research is needed to ascertain whether the main findings can be generalised to a larger region or over a longitudinal data set. The focus on a specific period also limits our ability to see potential time trends in these indicators and their relations. For example, it may be more informative to study “Walk score” over time to see the impact of universities on their surrounding environments. However, there remain challenges for collecting data over larger scales. Future research could also build on expanding the set of potential OKI indicators. For example, analysing the diversity of user groups that access the university website, library and publications may be useful. Further investigation is needed to examine the accessibility to such data.