Management Perspectives towards the Data-Driven Organization in the Energy Sector

Abstract

This paper explores the current attitudes of managers and executives working in the energy sector towards the Data-Driven Organizational Model implied by Big Data. The aim is to explore and understand the current mindset of senior decision makers, since their success depends as much on cognitive and behavioral processes as on their technical competences. We adopt a grounded-theory approach, developing models of understanding and belief abductively, driven by the data obtained from participants through a reflection guide. We find that managers differ significantly in their understanding and engagement with their challenges; they display interest but differ in their commitment and enthusiasm; they identify a lack of strategy and skills as current barriers; and they are currently unwilling to trust data, treating evidence according to their own prior commitments. This is a significant barrier to establishing the Data-Driven Organizational Model. These findings raise concerns, and the paper concludes that by considering initiatives for implementing more agile and forward-looking approaches, establishing a data-driven organizational culture, and managing such changes effectively.

1. Introduction

This paper explores the current attitudes of managers and executives working in the energy sector towards the data-driven organizational model (DDOM) that is implied by Big Data (BD) and Big Data Analytics (BDA). BD and BDA are now essential elements of their landscape and becoming data-driven is central to the digital transformation journey of these energy companies. More generally, BD and BDA have been accepted and absorbed into many mainstream companies. This is evidenced [1], for example, by the appointment of Chief Data Officers (65% of companies), data investment initiatives (99%), and reporting of measurable business outcomes (96%).

Our focus on the energy sector has two motivations. Firstly, this industry consists of mostly large and consequently less agile companies, for which any management change is subject to considerable inertia. Secondly, the energy sector is marked by a significant transformation required by the European Green Deal Action Plan [2], and determined by 3-D’s megatrends of decarbonization, decentralization, and digitalization [3,4], that force the companies to reconsider their long-term business models. To meet the goal of the EU core strategy, digitalization of energy systems will pay a key role [4]. According to Svenja Schulze, Federal Minister for the Environment in Germany, “Digitalization is an excellent lever to accelerate the transition towards a climate-neutral, circular and more resilient economy” [5]. To increase the impact of the digitalization in energy sector, which already adopted digital technologies for several years in the context of technical applications such as simulation, modelling for product design, monitoring, control, planning, markets, and forecasting [6], several key elements must be triggered and involved as parts of the game: Internet of Things, big data and predictive analytics [4]. As defined by Gartner, the concept of digitalization means the use of digital technologies to change a business model and provide new revenue and value-producing opportunities [7]. Having clarified these matters, we consider that a data-driven transformation may enable a company to achieve digitalization, regardless of the goals that are assumed or imposed by rules and regulations [8,9] or defined at the level of the organization’s strategy.

Thus, any company, and especially energy companies, facing obligations to meet targets under the EU Green Deal Action Plan will need a coordinated strategy that systematically addresses the 3-D’s megatrends. From the perspective of this paper, it is digitalization that is of concern and needs to be investigated; it is essential that business leaders are well placed for appropriate action and decision making. Established beliefs, however, still tend to stifle progress; the understanding and mindset of managers and decision makers will have a crucial impact on what can be achieved. The success or failure of any organizational change implied by new information technologies such as BD depends, above all, on people’s attitudes towards it. Moreover, becoming a DDOM crucially depends on people’s understanding of it. Despite the widespread acknowledgment of its importance, real progress towards becoming a DDOM remains a significant challenge.

Among the challenges raised by BD and the data-driven paradigm for the business environment is its response to a new and relatively unexplored source of value; the value of “insight” that emerges from data. Thus, the paradigm confronts and almost seduces businesses with the promise of these new and valuable insights. At the same time, it can weaken the decision-making role of the decision maker without, however, reducing their accountability for the decisions taken. As a result, social and especially ethical issues emerge.

Thus, the implications for education and professional development are considerable; as the role of the manager changes in a BD world—specifically their position, functions, and responsibilities—the education provided to them must be constantly adapted.

Fundamentally, the understanding and beliefs of business leaders is a major limiting factor in achieving the transformations necessary to affect essential change. In the energy sector the importance of these changes is underpinned by the green agenda and its manifestation in the EU Green Deal Action Plan requirements. Clarifying these matters in the current context, which is the subject of this paper, is therefore imperative.

Since the topic cannot be approached with clear assumptions or a theory a priori, we must adopt an appropriate research methodology. The approach is abductive; we choose a grounded theory strategy in which, through data obtained from the constituency of participant managers and executives from the Romanian energy sector through a reflection guide, we build models of understanding and belief driven by their responses to open-ended questions. This is discussed in detail in Section 4, below.

This research is situated in the context of the potential transformation as described above. More specifically, it aims to answer the following questions:

RQ1. What do managers think about the current and potential impact of Big Data, Big Data Analytics and the Data–Driven Approach on their businesses?

RQ2. How do managers perceive the role of data in the decision-making process?

RQ3. To what extent do managers feel that the Data-Driven Organizational Model is currently deployed in their businesses?

The remainder of this paper is divided into four sections. Section 2 comprises an overview of DDOM concepts and practices as they have been addressed in the literature. The next section explains and justifies the research methodology and then moves on to the results and analysis of our findings. The paper ends with conclusions, reflections and recommendations, limitations of the current study, and on directions for further investigation.

2. Literature Review

This section focuses of on a review of the concepts specific to the research domain: big data (BD), big data analytics (DBA), the data-driven paradigm, and the management perceptions and attitudes concerning these issues. These topics are considered here as they appear in both academic and practitioner literature targeting energy sector.

2.1. Background

As discussed in the introduction, transformations in the energy sector are driven to a significant extent by the EU Green Deal strategy and the “three-Ds megatrends” of decarbonization, decentralization, and—most importantly from our perspective—digitalization [10].

The EGD Agreement provides a strategic plan for the development of a sustainable EU economy, with actions to boost the efficient use of resources, highlight the necessary investments, and provide the available financing instruments [11,12]. The energy DSOs (Distribution System Operator) would be at the center of this energy transition evolving towards a new digitally operated and distributed energy system under the concept of smart grids [13,14], and facilitating the market in a neutral manner.

Various researchers noticed that big data gain tremendous advantage in the last years and the technological developments in the big data domain have increased the market need for developed management skills using knowledge and perceptions from metadata to create profitable business models [15,16,17,18].

A considerable number of articles have addressed the “big data movement” that has the ability to extract “intelligence from data” and generate valuable information and convert it into valuable advantage to improve business performance [19,20,21]. Moving from traditional analysis to big data using the initially 3 V characteristics (volume, velocity, and variety) [19,22], is now enhanced with other levels introduced by researchers and big IT corporations, like IBM, which brought in the fourth V (veracity) as the level of accuracy in some sources of data, SAS with the fifth V (variability) (and complexity) of data and sources, Oracle presenting the sixth V (value) of big data [23,24,25], and the seventh V(visualization), defined as interpreting the patterns and trends that are present in the data [26,27].

Increasing digitalization and automation of the infrastructure, especially for the energy sector, are essential to process “high-volume, high-velocity data” considering the characteristics of big data scenarios: sensors, communication, computation, and control capabilities. In this way, the information can be offered in the form of “service and maintenance reports about regular and unexpected repairs, health sensor data from self-monitoring assets, data on end usage and power feed-in from smart meters, and high-resolution real-time data from GPS-synchronized phasor measurement units or intelligent protection and relay devices” [28].

Taking into consideration the above, at the level of an organization BD can be both a resource and a capability [16,22] pointed out that investing in big data resources is sometimes not enough. There is the need for a dynamic business intelligence system with capability to process, manage, and deploy the resources, enhancing the performance of the organization by leveraging of big data resources for sustainability and development [16,29].

Modern organizations increasingly invest in BDA projects to reduce costs, make accurate decision making, and future business planning. The BDA is used in business sectors like retail, telecommunications, hospitality, entertainment, transportation, food service, insurance, finance, manufacturing, and online businesses, which are successful examples [30]. Because the structure, format, and quality of data is very wide and in massive volumes, the big data from energy sector are collected from many different sources using different data acquisition devices and with an integration of databases vital for the BDA tasks [31].

According to a report provided by Mordor Intelligence in 2022, BDA plays an essential role and many corporations adopted it in reducing energy consumption and improving energy efficiency. Between problems encountered in this sector the report mentioned “predicting production demand, enhancing efficiency, load distribution and optimization, and optimizing operation processes” [32]. Nevertheless, BDA provides many enhancements for the end users.

In 2015, the Capgemini report [4,33], showed that 80% of the utilities considered Big Data analytics as a source of new business opportunities but only 20% have implemented initiatives in this area. Several difficulties were holding them back: data complexity, access and privacy issues (54%), data storage and manipulation costs (26%), and skills shortages (13%). In 2021 the Capgemini report, with the aim to also meet the requirements of the EU Commission regarding ‘Industry 5.0′ emphasizing DSO, which refers to people working alongside smart assets, proves the new priorities of the century: “An economy that works for people”, “European Green Deal”, and “Europe fit for the digital age” [34]. The 23rd edition of World Energy Markets Observatory (WEMO) revealed that “It’s about enabling assets to work better and faster by leveraging advanced technologies, including integrated technologies via the IoT and big data, automating the asset management process with real-time data from the field to create the ‘Ultimate Smart Grid” [14].

2.2. The Data-Driven Paradigm

Organizations face the need for complex data analysis (both structured and unstructured) to cope with the competitive landscape in the marketplace. The economic value of data as well as its strategic management are elements that can no longer be ignored [35].

Organizations are turning to advanced analytics (predictive and prescriptive) to discover and capitalize on new business opportunities and insights [36]. Traditionally, data analysis was based on descriptive analyses that provided an explanation and interpretation of past events. The managerial paradigm has now changed so that the focus is on predictive and prescriptive analysis. Predictive analytics allows the identification of previously unknown patterns in an organization’s data, using data-mining tools. Prescriptive analysis provides organizations with an optimal decision given a complex set of objectives, requirements, and constraints with the goal of improving business performance [37,38].

In a data-driven company, data are the core of the business, and its decision-making processes rely primarily on it. Many organizations consider themselves as being data-driven because their decision makers look at evidence, form an opinion, and then make their decision. However, these kinds of decisions are only “data-inspired”, meaning that they are made based on some data and use immediate knowledge, automation, and intuition to reach a conclusion [39,40].

The same idea is supported by Ciasullo [41], arguing that “by going beyond both intuition and experience [19,42], data become a driving force to inform actions, predict complexity and foster change [43]”.

2.2.1. The Data-Driven Business Model

The existing literature around business models has evolved significantly during recent years and the concept is now used in the context of e-business, strategy, and innovation management [38,44].

One of the first business model frameworks was provided by [45]. They characterize the concept of a business model by the functions it fulfils. A business model articulates the value proposition of a company, “the value created for users by the offering based on the technology” [38].

Johnson et al. [46] proposed a business model framework consisting of four interlocking components: the customer value, the profit formula, key resources (technology, people, brand or partnerships), and key processes. Based on a review of existing business model literature, Osterwalder [47], synthesized a business model framework consisting of nine building-blocks, namely, value proposition, key processes, key resources, key partners, customer relationships, channels, customer segment, revenue streams, and cost structure [38].

Osterwalder and Pigneur [48], describe in their vision “A business model describes the rationale of how an organization creates, delivers, and captures value”.

This trend of generating value from data has led to a new paradigm-data-driven business model (DDBM) [38,49]. Departing from traditional business models, which consider data as a resource, DDBM relies on data as a key resource to enhance value creation and instill value [38,50]. In other words, in a DDBM, data are considered a strategic asset which requires not only an increasingly skillful use of data, but also a cultural change in the mindset of managers [41].

Hartmann et al. [38] developed a framework that allows for a systematic analysis and comparison of DDBMs. The dimensions of DDBM framework were derived from a systematic review of the most important existing business models (including the work of [46,47]). The DDBM model is structured into six key dimensions: key resources, key activities, value proposition, customer segment, revenue model, and cost structure [38].

There are four main characteristics of a DDBM identified by Fruhwith et al. [51]: data used as a key resource [38,50,52], customer value from data generated by data analytics activities [38,53], and data or information that is part of the value proposition [38,52] and can be monetized [53,54].

Brownlow et al. [55] propose through the DDBM blueprint, a solution for transforming an organization into a feasible data-driven organization, starting from six fundamental questions: “What do we want to achieve by using big data? What is our desired offering? What data do we need and how will we acquire it? How will we process and apply this data?, How will we monetize it? and What are the barriers that prevent us from achieving our goal? [55]” This model is focused on desired business outcomes, organizational dynamics, resources, competencies, and the business sector the organization falls into. The applicability of this model can be found in areas such as telecommunications, retail, and financial services (e.g., Zara, AT&T, ING Direct, Goldman Sachs etc.).

Lange et al. (2020) [30] developed a simpler four-step DDBM process (analysis, design, implementation, and review), which is nevertheless mapped onto key characteristics of DDBM models found in the literature [38,55], as discussed above.

2.2.2. Management Perceptions and Attitudes concerning BD, BDA and DDBMs

The technical challenges of using big data and data analytics are very pertinent (real), but the changes in managerial perception are even greater, starting with the role of the leadership team [19]. Active involvement of senior management in developing a strategy, with clear goals for establishing a data-driven culture aligned with the business strategy, is a key element in implementing a data-driven business [36].

NewVantage’s 2021 [1], survey shows that while respondents to the survey, 76% of whom are data/analytics executives, are positive about the technologies they manage, they are less enthusiastic about their own roles. There seems to be a growing consensus within organizations that the Chief Data/Analytics Officer has primary responsibility for data, 49%, but a quarter of companies still report that there is no single point of responsibility.

The power of big data does not erase the need for human insight or understanding [19]. In the context of a data-driven business, intelligence derived from data will not generate the expected performance if managers fail to foresee the potential of newly extracted information [29]. In this regard, it is necessary for managers to have the necessary skills to ensure that they understand how and where to integrate extracted information based on data.

The transformations demanded by data-driven businesses require from big data managers an ability to understand the current needs and predict the future needs of other business units, customers and other partners [29]. In addition, mutual trust and a positive working relationship between big data managers and other functional managers is likely to lead to the development of superior big data people skills.

Although emerging technologies enable advanced data analytics, without a solid data-driven strategy, the organization’s business goals can be affected by inaccurate forecasts, unnecessary costs, wasted time, poor decision-making processes, low profitability indicators. Data-driven strategy must be aligned with business strategy [35].

Another management issue is trust in data-driven data. A KPMG study [56], points out that managers understand at a general level the value and potential data are providing, but often are reluctant in what actually are data telling them. “25% of executives indicate that they have limited or no confidence in their data. Further, 2/3 have ignored data-generated information to make decisions based on their own intuition”.

Barriers to data-driven business adoption are managerial and cultural rather than data and technology related. “The main barrier to the widespread adoption of analytics is the lack of understanding of how to use analytics for business improvement [42]”.

The first five elements provide a focus for leadership attention: data quality, using analytics to improve decision making, creating a big data and analytics strategy, making data available, and developing data literacy within the organization [57]. Improving the quality and credibility of data is a key enabler and potentially a barrier to value creation from big data and business analytics.

McAfee and Brynjolfsson (2012) point out that “companies in the top third of their industry in their use of data-driven decision making were, on average, 5% more productive and 6% more profitable than their competitors”. The authors’ conclusion is very clear: “Data-driven decisions tend to be better decisions. Leaders will either embrace this or be replaced by others who do [19]”.

All factors that influence business decision making by leaders are mainly determined by two main components, data-driven decision making and psychology-driven decision making [58]. Uncertainties and dynamic business conditions have increased the complexity and ambiguity of decision makers. The authors believe that VUCA (Vulnerability, Uncertainty, Complexity, and Ambiguity) conditions need intuitive capacity in decision making. These complex and uncertain conditions in which the business environment evolves today has changed the mindset of global leaders and business decision makers towards increasing adaptability, agility, and developing strategic planning based on VUCA characteristics [58].

In data-driven business, managers need to have ambidextrous capabilities, in other words they need to combine both intuitive and data-driven analytical skills [58]. The results of the study provided by Elfindah and Wibowo confirm that both analytical and intuitive ability are inseparable. The same idea is also found by Gupta and George [29], who point out that it is important to strike a balance in the decision model between being analytical and being intuitive and, moreover, to see the complementarity between these two characteristics.

3. Research Methodology

Our study of perceptions takes an interpretive approach that is particularly suitable to exploring human experience and constructing reality through the meanings that participants ascribe to that experience. Rather than verifying the application of a certain pre-existing theory in a certain business context, we take a grounded-theory approach to analyze, interpret, and explain the meanings of the social actors involved in the study [59,60], and thus to contribute to developing theoretical explanations of phenomena that occur in specific contexts.

The participants were asked to provide their responses to open-ended questions that focused as much on value judgments (their opinions and perceptions) as on the elicitation of factual material. Indeed, we had no a priori expectations that their self understanding of the terms we are considering was correct, or that they themselves recognized the limits of their understanding. As such, a methodology that develops theory abductively driven by the data and leading to categories and classifications that emerge synthetically, is best suited to our needs.

Our topic of investigation and our research questions are highly context-dependent in both time and space; the landscape is changing rapidly, and the actors and institutions involved are developing approaches and their understanding as this change is taking place. The reflection guide comprises questions that explore these developments and competences as they evolve. It was developed strictly in line with the underlying research framework; this ensures that the data collection is well structured and methodical, while allowing participants to describe their own and their companies’ practices, and to construct personal interpretations generated by the circumstances that prevail in their varied contexts. As such, an interpretative approach in which sense and meaning is primarily driven by the actors is not only appropriate but essential.

The classifications and categories that emerge in Section 4 were generated from the free-text responses of participants to the reflection guide. The sense in which this methodology is abductive [61], is captured by the processes of inductive synthesis from data and the deductive testing of tentative analyses that emerge from that synthesis against further such data. The results, as reported in Section 4 are underpinned by the merging of these two processes over the entire data set.

The target population consists of managers and executives from Romanian companies in the energy sector. Detailed responses from 24 respondents were collected from six different companies.

Appropriate ethical standards have been adopted: the anonymity of the respondents was ensured, the data were collected on the internet, and no pressure to participate in this study was applied.

4. Results and Discussion

The data on which the following analysis and discussion are based comprise the responses to the Reflection Guide (see Appendix A). There are seven questions in this guide, and these are aligned with the three research questions that were provided in Section 1, above.

The high-level overview of results provided below in

Table 1. Research framework-Topics investigated.

Research Question	Theme	Sub-Theme	Reflection Guide Question
RQ1. What do managers think about the current and potential impact of Big Data, Big Data Analytics and the Data–Driven Approach on their businesses?	Awareness and business impact of Big Data, Big Data Analytics and the Data-Driven Approach	Concepts	1,2
		Impact
		Opportunities and Advantages
RQ2. How do managers perceive the role of data in the decision-making process?	Data in the context of the decision-making process	Data confidence	6
		Data versus experience and intuition	7
RQ3. To what extent do managers feel that the Data-Driven Organizational Model is currently deployed in their businesses?	The data-driven organizational model	Achievements	2,3
		Restraining factors	4
		Action Plans	5

introduces themes and sub-themes that emerged synthetically from the raw data obtained in line with the grounded-theory research methodology.

Our analysis of the data follows the usual format for grounded theory. As patterns emerge in the raw data through repetitions and similarities, these are categorized through tagging. These tentative codes are subject to revision or adjustment as further data are analyzed. It is in this sense that the analysis is abductive: the final codes, which using the standard terminology of the methodology, are–finally–concepts, once the interplay of bottom-up data and top-down classification has been completed. These concepts are then further classified into categories: once again, the standard terminology of the methodology.

Full details of the low-level analysis of coding and grouping not provided here; however, through representative examples of participant responses, indicative support for these classifications are provided. Verbatim quotes (the responses were always in English despite this typically not being their first language. Grammatical and spelling infelicities have been kept.) from participants are shown below in italics.

In what follows the three major subsections of this section of the paper align to the main themes relating to the research questions that are identified in Table 1. Within these subsections, we cover the corresponding sub-themes, again as shown in Table 1. This section of the paper ends with a discussion of the model overall.

4.1. Big Data, Big Data Analytics and Data-Driven Approach-Awareness and Business Impact

4.1.1. Concepts and Awareness

We are interested to see if executives are aware of BD and BDA and also if they see the benefits of implementing these new technologies in their organizations. We investigated what is really new and important about BD from a business perspective and how the data-driven paradigm is understood and perceived at the management level.

The responses show that executives are generally fully aware of BD and BDA, the most common sources of information being, “self-learning”, “training inside the organization”, or “IT companies’ presentations”. Big Data and Artificial Intelligence are considered as being new trendy subjects when it comes to technologies. According to one respondent, “Everyone talks of big data and AI, so it is impossible not to become aware of it”. All the respondents have become acquainted with subjects relating to BD and BDA from different business or professional contexts. (Awareness, Strong)

However, some of the responses reveal a degree of confusion regarding the meaning of the Big Data concept itself. The most common misunderstandings relate to the nature of the data, in terms of necessary accuracy and its degree of structuring. Some inaccurately consider Big Data as being part of Business Intelligence. As one respondent notices: “The Big Data is the fundament of all BI (Business Intelligence) results which create the opportunities models and actual performance”.

The data-driven concept seems to be less well acknowledged. While, quite accurately, all respondents relate this concept with the decision-making process, the majority consider that data-driven means simply relying more on data and in larger amounts in decisional contexts. This appears in several responses:

“Data—driven is a new way to support decisions, by providing more meaningful data and more intelligent technologies to make sense of it” or

“Data is the blood in a company having the goal to digitalize more and more of it’s activity. To achieve that target the first and most important step is to standardize data and to create data warehouses that would add more value to the business. I became aware of this organizational model now when digital, AI, RPA, cybersecurity are being evaluated in the KPI’s in competitive companies and not having all or part of them in the top strategic directions would be a great risk for the going concern of that company.”

(Awareness Weak)

Only few respondents mentioned the differences in both data types and ways to exploit it, as well as the implications for management processes. No respondent mentioned the leading role of data in the decision-making process.

This suggests that action is needed to increase the understanding of these new concepts and technologies at management levels.

4.1.2. The Business Impact of Big Data and the Data-Driven Approach

The value and advantages of Big Data are fully recognized. These opinions appear in several responses:

“We can’t afford to ignore this important source of data when we take decisions” or

“Nowadays, the insights provided by Big data make the difference between being informed or uninformed when it comes to manage a business”

Opportunities and Benefits

Generally, companies evaluate Big Data opportunities systematically when they formulate or extend their business intelligence strategies. As one respondent noticed, “Big data have the benefit of being the fuel for digital tools, that could present instant actual reports and statistical data on the performance of the business, that could feed RPA’s in repetitive tasks, that could highlight trends and models supporting decision making”.

The respondents fall into three distinct groups based on their expressed attitudes towards the value of becoming a data-driven organization: interested, skeptical, or enthusiastic.

The managers from the interested group consider Big Data an important business opportunity and usually see data-driven initiatives integrated within their Business Intelligence infrastructure. Their responses suggest that there are two main reasons why Big Data and data-driven initiatives represent an opportunity for businesses: “the need to capitalize on huge volumes of complex unstructured data” and “the need to build on predictive and prescriptive mechanisms”. As main benefits they identify, “new insights related to the customers’ behavior and expectations”, “ability to predict the future”, “better risk mitigation”, “increased revenues”, “more efficient operations”, “improved control of operation processes” and “faster product innovation”. (Interested)

Other respondents tend to propagandize Big Data, considering that Big Data and its analytics offer a lot of opportunities. They are part of the enthusiastic group–mostly managers with an IT background that are trying to push the adoption of these new technologies. As two representatives of this group indicated, “I am very interested in new technologies and their potential business value–especially Big Data and Artificial Intelligence-and I promote them throughout my company” and “The ability to exploit big data properly could significantly change the way we do business.” The enthusiastic group is represented mostly by those that overrate new technologies and use visionary language to express their potential benefits. These attitudes appear in several responses: “Big Data will revolutionize the way we do business” and “Big Data will dramatically change our business processes and operations in the near future”.

Propagandizing Big Data, together with the fact that innovations in technologies play a predominant role in IT–Business relationships, make it hard for business professionals to identify their real opportunities, appreciate the risks and formulate rational strategies. Managers should be able to clearly and very specifically assess BD and BDA as opportunities for their businesses. Otherwise, the expectations relating to these technologies are unrealistic. (Enthusiastic)

Other managers do not see Big Data as a notable opportunity for their business. Generally, they claim that their companies do not generate unstructured data in the normal course of their business and therefore that BD is not likely to bringing any value. For example, one respondent claims, “We are not a digital company and we are not interested in using Big Data or Business Analytics”. The managers who do not regard Big Data as an opportunity for their business do not identify potential benefits either–they are part of the skeptical group. Mostly, they raise questions related to the viability of eventual investments in data-driven initiatives. As one respondent commented, “We should be able to quantify how much using the data-driven insights impacts the bottom line before investing in Big Data and in the required technological capacity.” These managers would need to clearly identify the potential value of the insight that can be generated from Big Data and only afterwards consider including these technologies in their IT infrastructure. (Skeptical)

Regardless of their declared affiliation to one or other of these groups, all respondents were genuinely interested in the results of this study.

Sustainable Development

The impact of Big Data and data-driven business models on organizations’ sustainable development is largely recognized. As two respondents indicated,

“DDOM [the Data-driven organizational model] would allow the company to target and monitor sustainability KPIs which will enable the company to attract green financing (green bonds, sustainable linked loans, etc.)” and

“They [data-driven business models] can have an impact on sustainable development when used for reports on carbon footprint or even when used for a digital data flow instead of paper based one”.

See Figure 1, below.

4.2. Data in the Context of the Decision-Making Process

Irrespective of the managers’ degree of understanding concepts like data-driven or the data-driven organizational model, the simplest way to assess if a company is data-driven or not is through its decision-making processes. Therefore, we were interested to investigate the attitudes of managers regarding the balance in the decision-making process between experiences and beliefs on the one side and the evidence provided by (Big) Data on the other. (Priority)

4.2.1. Data Confidence

Trusting data and the information provided by sophisticated and recently adapting and auto-updating algorithms is a sensitive topic. We are used to learning to trust other people, based on common experiences and we are also used to trusting ourselves, our gut instincts, and actionable knowledge, based on our expertise and experience. With all the reassurances that we receive from BD and BDA professionals that relate to our unique and irreplaceable role in the decision-making processes, it is difficult to learn to trust data in the absence of a prior testing process.

The majority of the respondents are aware that data are the primary source for the information needed in the decision-making process. Two of them provided well-known quotes as answers: “[Without] data, you’re just another person with an opinion (Edwards Deming)” and “In God we trust, all the rest bring data (the mantra of the modern manager)”.

However, most of them question both data quality and data accuracy—an aspect that is mainly related to the quality of their existing information systems. It is normal that all the disappointments related to their current IT infrastructure influence their attitude towards any new IT project.

Generally, managers consider that people play the active role in effectively making sense of data. “Data is giving you just a hint–you should know very well your problem and fully understand that data, otherwise you can’t make sense of it”.

Few respondents do not trust data in the decision-making process. “In my experience, one should be suspicious about data and always question it”.

4.2.2. Data Versus Experience and Intuition in the Decision-Making Process

One of the more critical aspects of Big Data is its impact on decision making. It can weaken the decision-making role of decision-makers, without reducing their responsibility for those decisions. We were interested to see to what extent the managers are openly data-driven and ready to disregard their own intuition if the data should contradict it.

We found that managers are not ready to trust data fully and to let it replace, even partially, their judgments or intuitions in a particular decision-making context (Priority, Weak):

“Data offers only an indication–the decision is a result of intuition and experience” and

[If data contradicts my intuition I would] “Challenge the data and the algorithms that generated the contradiction.”

However, some respondents pay much more attention to data in decision making. If data does not validate their judgment based on experience and intuition, they search for additional data to support their decision. For example, one respondent stated, “If data would contradict my gut feeling I would search for more data to include in the analysis”.

Some are willing to fully base their decisions on evidence, but only after double checking it in case it challenges their opinion. “I would verify again the data and in case its confirmed, I would change my opinion based on the data.” (Priority, Strong).

See Figure 2, below.

Respondents are generally partisans of the cognitive approach to decision making. However, they look for data to support their actions; but in these cases, the most common approach is to question the data rather than their judgments. We may conclude that managers advocate for a data-inspired approach rather than a data-driven approach to the decision-making process.

The general lack of trust in data accuracy and quality is another aspect that requires attention. This concern is manifested by most respondents:

“We need to make sure that the data is validated to avoid a garbage in-garbage out effect”

or

“The data is not there to confirm my expertise, but to highlight important aspects. Still, it must always be checked for correctness”.

Big Data does not need to be utterly precise and accurate, but the managers’ attitudes towards data reflect their experience and frustrations relating to the data that they currently use (structured data, stored in organizations’ data bases, and data warehouses). Therefore, since Big Data may be imprecise, it is quite normal to be comfortable with this imprecision or even ambiguity at the micro level, as it pays off in valuable insight at the macro level. However, it is important to address the right data in a particular decision context. It could be useful to raise awareness throughout the companies of this important and generally less-acknowledged aspect.

4.3. The Data-Driven Organizational Model

By examining this topic, we aimed to explore the opinion of executives regarding the DDOM and the current stage of data maturity of their businesses.

4.3.1. Achievements

Organizations regard themselves as being at different stages of transition to a data-driven model, with most of them at the very early stages.

However, the responses suggest that the DDOM concept is not completely understood (as discussed in Section 4.1 above). Generally, managers focus on the amount of data generated, or on the sophistication of their IT infrastructure, when they characterize their companies’ data maturity. Some respondents identified their current information systems—that include only structured data and information—as data-driven. This position appears in several responses: “We use reports provided by our Management Information Systems in the decision-making process. As these reports contain important data, I would say that my company is a data-driven organization” and “We use a lot of evidence to fundament our decisions. Especially dashboards that allow us to see what is happening in real time.”

Most of the of the respondents consider that their organization is not yet data-driven and, moreover, that there are no appropriate strategies in place to address this issue. Other managers consider that becoming data-driven is explicitly part of their strategy. “We are not yet a data-driven company. It is a matter both of developing the system and validating the existing data, but it is part of our strategy to digitalize the business as it is needed considering the changes of the energy market model as part of the Green Deal and Fit for 55 initiatives”. However, some executives describe their organizations—or at least a part of their business models—as being data-driven. As expected, data-driven approaches are more present in customer-facing business processes, “We use big data analysis to know better our customers and we use the results in marketing campaigns”.

Almost all respondents claim that they are interested in moving towards a data-driven organizational model, but there are no clear guidelines or best practices that they can share regarding this mission.

4.3.2. Addressing the Challenges of Implementing a Data-Driven Organizational Model

Under this theme we investigated both the main challenges raised by Big Data and the data-driven approach and ways of addressing these challenges from a managerial perspective.

Restrictive Factors

BD and the data-driven paradigm raise new and varied challenges for businesses. In order to address them, organizations should clearly identify and analyze all these challenges and prioritize them.

According to the respondents, the main barriers to a DDOM are:

• Leadership–lack of vision, lack of commitment (Vision, Inertia);
• Lack of skills (Skills);
• Business volatility (Volatility);
• Lack of employees’ commitment and support (Support).

Other identified limiting factors refer to the data itself (data quality) and lack of resources such as “funds, time allocated for this project”.

The main challenges organizations face in their effort to become data-driven are a lack of strategy and a lack of skills. This is not surprising, as in these types of transitions, organizational and human issues are generally considered to be more important than the technical. As such, this is the major motivation for this research. Any initiative to implement or to enable a data-driven culture within organizations must consider the attitude of managers relating to this subject. Do they fully understand what such an initiative means? Do they see the opportunities? Do they have the right mind-set? Are they ready to promote a culture that values data in their company? These are important questions that any company should have clearly answered before considering any other technological or infrastructural aspect involved in the process of becoming data-driven. Data-driven business transformation is a long-term process that requires focus, commitment, and persistence; without executives’ full commitment and support it cannot be a successful endeavor. The shortage of data specialists (data scientists, data translators) remains also one the most important issues for most organizations.

Plans for Action

The responses indicate that executives are willing to address the previously identified challenges. The vision of the respondents regarding a data-driven strategy can be summed up in the following actions:

• Understand the potential benefits for the organization-setting the right expectations (Vision, Focus);
• Conduct change-management activities (Vision, Focus);
• Develop human resource policies to attract Big Data specialists (Skills);
• Provide Training for managers and business professionals (Social, Skills).

One interesting observation is the occurrence of both vision and change management as important issues in implementing a DDOM. As the respondents are managers—most of them occupying senior positions—this opinion represents a significant result. Another interesting observation is based on this response, “The major problem is that we lack data scientists. However, we should be aware that we also need people to bridge the gap between business functions and the technology department–data translators or citizen data scientists” This is a well-made and relevant point. Any project that involves a technological change at the organizational level requires, first of all, the existence of a solid information partnership between business users, analysts, and IT professionals (Social).

The main actions the managers identified as possible organizational responses to the challenges raised by Big Data and the data-driven approach are the need to involve everyone in change management activities and to hire big data specialists (data scientists, data translators). One important result refers to the need to train executives in order to develop the necessary skills to effectively manage data-driven business processes.

See Figure 3, below.

4.4. The Emergent Theoretical Model

Our final model has six emergent categories which are the major conceptual groupings. Associated with these are 15 concepts. These relate directly to the categories but, importantly, both categories and concepts overlap and relate to one another indirectly. Thus, the overall conceptual model is a graph rather than a tree of ever coarser-grained conceptualizations. These relations and correspondences are not immediately evident from the direct responses of participants and are the result of the grounded-theory analysis.

See Figure 4, below.

Our findings in this section will be summarized next, where we also discuss their significance in relation to the general trajectory that these energy companies need to follow; namely, successful digitalization that informs the related targets of decentralization and decarbonization, as required by the EU Green Deal Action Plan.

5. Conclusions

As discussed in the introduction, we argue that the cognitive and behavioral aspects of change management driven by BD and the DDOM are critical limiting factors to success and dominate the technical challenges that are naturally also centrally implicated. Companies that systematically misunderstand, underestimate, or lack conceptual skills, face barriers that cannot be overcome by recourse to technical infrastructure or expertise. Fundamentally, success or failure of any organizational change prompted by new information technologies depends primarily on people and their attitudes.

As is clear from Section 4, across the companies surveyed, a complex picture of these cognitive and behavioral processes has emerged. These are large energy companies across one EU country and constitute a substantial proportion of the sector. They carry, therefore, a significant national burden regarding meeting EU directives, particularly in the area of climate change and specifically the European Green Deal Action Plan and the Fit for 55 packages [62]. Success in digitalization, and consequent success in decentralization and decarbonization, is crucial for these companies and for the country. The conceptual model we have developed suggests that this may be hampered by significant non-technical limitations.

5.1. Recommendations

First, our model indicates there is evidence of fundamental conceptual confusion regarding BD and the DDOM in some areas.

All participants claimed awareness of BD and BDA and are interested in their potential. However, their level of understanding differs, as does their overall attitude, which ranges from skeptical to potentially overly enthusiastic. The concept of data-driven is generally less well understood than is the concept of BD itself. Their view of the potential impact varies greatly in extent and focus. This is captured in Figure 1 in Section 4.1.2. The relationship between the three categories of Awareness, Approach, and Impact is subtle: assessments of Approach and, especially, Impact, are critically related to Awareness. However, Impact is to some extent directly driven by participants’ very clear understanding of their responsibilities in sustainable development, and the importance of digitalization as a major theme in their commitments under EU directives such as the European Green Deal Action Plan.

These results strongly suggest that companies need to adopt consciousness-raising activities and training within their organizations to ensure there is an accurate understanding of what BD and BDA offers and what the DDOM consists of. Such improved understanding will then feed into more refined views on the Approach and realistic judgments of Impact.

Of course, instituting such activities and training itself depends on sufficient self-knowledge. It is worth noticing, in this context, the authentic interest of executives in our research. Almost all respondents specifically asked us to share with them the results of our research, perhaps anticipating the extent of their own limitations.

Second, there are questions relating to the direction of change where data-driven activities are currently being pursued. It is in this area that weaknesses in understanding the DDOM itself is most critical. These companies are generally dealing with ever increasing amounts of data, and they concentrate on this, relying on standard Business Intelligence approaches, such as dashboards, and seeing this as, to some extent, data-driven. There is also a recognition that there is much more than can or should be done, and that there is a lack of a systematic organization-level strategy to address this. Organizations appear to already understand that there are leadership deficiencies, skill gaps, and a lack of commitment and support—the last of these the result of a lack of strategic and resource focus. It is also pleasing to see that these recognitions are to some extent accompanied by relevant ideas to address these challenges. In organizations whose Approach is largely committed (enthusiastic and interested) our conclusions in this area are largely already acknowledged.

To address the DDOM, all these organizations need an explicit and resourced strategy with a clear vision and focus, to establish an integrated approach. This needs to be accompanied by an aligned program of appropriate change management and systems for monitoring and measuring results. To address weaknesses in expertise, prioritizing the recruitment of BD specialists is needed. Not only will this strengthen the technical constituency, but such expertise will also support, moderate, and inform non-technical leaders appropriately, in terms of possibilities and expectations. Finally, building on the training discussed earlier, specific leadership training for managers and executives should be in place to complement the strengthening of the technical base. Leaders need to ensure that data-driven innovation is a priority, implement strategy, and manage change, informed by a clear understanding of the DDOM.

Third, there are misunderstandings and attitudes that relate to decision making in the context of BD and BDOM; in particular, trusting data in driving decisions in favor of reliance on managers’ intuitions and personal expertise. This is almost exclusively a matter of Trust and is captured in Figure 2. in Section 4.2.2. At present, this may be the deepest challenge for those responsible for decision making within these companies. There is a paradigm shift in moving to BD and the DDOM; the role and priority of data in decision making—particularly in relation to prior expectations and experience of those whose role it is to direct and implement organizational strategy overall. As we have discussed earlier in this paper, but which did not appear at all in any responses, there are associated matters of business ethics and matters of professional accountability associated with this radical shift. These are deep and important matters which do not, therefore, appear in our conceptual model. They do, however, need to be dealt with explicitly.

At some point in moving wholeheartedly to the DDOM, this will need to be better understood and recognized in practice. While this is a fundamental matter of importance, it cannot be addressed as a separate topic, and will need to be integrated into all the other recommendations for the actions discussed above, particularly and appropriately into the training of employees at all levels, especially for leaders.

5.2. Theoretical and Practical Implications

From a theoretical standpoint, this research does reinforce the productivity of the research methodology we have adopted. The fact that it was possible to construct a model from the free responses to the Reflection Guide strengthens our confidence in Grounded Theory as methodology that is not only well founded in theory, but is also workable and genuinely productive in practice. In the next two subsections, we discuss the possibilities of surveying participants at multiple levels of an organization; this would both address potential limitations in our research as well as allowing a deeper analysis and a more refined model. Such a multi-level analysis and multi-level modelling would require some adjustment to the approach—and would have interesting methodological implications for Grounded Theory, itself.

From a practical perspective, the results of research such as this can usefully be harnessed to help inform the development of strategy, change management plans, and training, as discussed in the previous subsection. As mentioned, the fact that the participants wish to be informed of our results is a good sign. At this level of generality, and in terms of practice, the limiting factor is the degree of detail in the model that is developed. As such, the idea of multi-level analysis discussed above, would provide such increased detail and, thus, permit more refined recommendations and practical impact.

5.3. Limitations

Two limitations of this work are to some degree evident. First, the number of respondents, while representative of their role, could usefully have been greater. However, there are clearly a limited number of relevant role holders at their level in the six organizations sampled. Six businesses may also appear a limiting factor; however, there are, of course, only a small number of businesses in the energy sector in Romania and, as such, the six surveyed provide excellent coverage.

Of more significance, perhaps, is the extent to which these senior managers—while responsible for strategic decision making—fully represent the institutional knowledge and competencies of their companies. It is possible that those in more junior positions have better or more refined understanding than their superiors and that, over time, institutional dynamics may trickle up critical knowledge and competences to those senior decision makers. This is discussed in the context of future research directions, below.

Second, in considering a broader constituency of companies, these energy companies are atypical. They are large and complex businesses with a good deal of verticality in their organizational charts. As noted in the introduction, they are likely less agile, displaying a degree of managerial inertia. Consequently, the paths through which institutional knowledge and competence (as discussed in the previous paragraph) may exist to permit sharing of best practice is likely more limited than in smaller businesses. Thus, it is not clear to what extent our results might be generally true of all relevant companies at this time. Alternatively, the results might well be generalizable to other large and complex businesses—particularly those that, like these energy companies, are subject to stringent external regulation.

5.4. Future Work

As discussed above, there are clear recommendations that arise from the conceptual model that results from the data collected. It is equally clear that these are themselves challenging to develop and implement. The process of transitioning to a data-driven organizational model is new, complex, and currently there are no best practices defined to guide it.

If the themes discussed in this article were explored further, and more deeply—for example by canvassing views from multiple respondents at differing levels of the organization and with differing responsibilities—more refined conclusions could be drawn, providing further guidance for the development of change management strategies.

In addition, an extended reflection guide that would allow us to further relate the categories that later emerge from the abductive methodology, allowing a better understanding of the relationships between baseline understanding and consequent opinions in other dimensions, would be beneficial. We already see a degree of a graphical structure in Figure 4 in Section 4.4, which to some extent reflects these interactions, but much more could be done with further questions and a larger constituency of participants with roles at all levels.

Other future research objectives include a more general analysis of how and to what extent business models need to be reconsidered in the Big Data era. Another intention is to explore the ethical and social issues relating to Big Data and to the datafication of the economic environment. As noted above, the business ethics and professional accountability matters that are associated with the shift to data-driven decision making, does not appear in our conceptual model. This is because current understanding of this area is so poor that no response provided relevant information. Eliciting attitudes, understanding, and current progress in these difficult areas are very important matters for future study, and are areas that we intend to pursue.