A Maturity Model to Determine the Degree of Utilization of Machine Learning in Production Planning and Control Processes

: The presented work introduces a maturity model for evaluating Machine Learning implementations, with a primary focus on Production Planning and Control processes, as well as broader organizational and technical aspects in companies. This model emerges as a response to the research gap identified in the analysis of 14 existing maturity models, which served as foundational bases for the development of this novel approach. By examining success factors and obstacles at different maturity levels, categorized according to defined dimensions and overarching design fields, this model can serve as a catalyst for bridging the research gap between models demanded in practice and the scholary exploration of topics related to Machine Learning in corporate processes. Notably, the structured design of this maturity model ensures accessibility for small and medium - sized enterprises (SMEs).


INTRODUCTION
With the rise of Artificial Intelligence (AI) technologies and applications, our society is witnessing a strong impact in day-to-day tasks through intelligent software solutions.Particularly prediction and generative models become more and more present in private and business environments.However, since most AI technologies require large amounts of training data, application-ready solutions are available mostly in domains where large amounts of (high quality) data is available.While this is clearly the case in internet-based information systems, where consumers voluntarily share and annotate texts, images, audios and videos, the situation is more difficult in industrial environments.Particularly in the realm of production planning and control (PPC) only few companies, are leveraging the potential brought about by AI and Machine Learning (ML).Theoretical research continues to push the boundaries of ML methods, while the industry, particularly small and medium-sized enterprises (SMEs), is primarily focused on implementing fundamental functions with respect to digitalization.Despite this, companies are eager to invest in AI disciplines such as ML due to the perceived importance of future business process enhancements.However, the realization of these benefits through practical implementation remains elusive.
A study of Gupta [1] shows that 78% of AI or ML projects do not reach a stage where they are ready for deployment in a productive environment.Fig. 1 shows the various reasons for early cancellation.In order to overcome such setbacks and improve the success rate of ML implementations, a systematic analysis of the ML-readiness and critical self-assessment and reflection is required by companies.This paper contributes to the state of the art by introducing a maturity model for companies to determine the degree of utilization of ML in PPC processes.To this end, Section 2 introduces the basic concepts required in the scope of this paper.Section 3 provides a systematic analysis of 14 existing maturity models, which results in the novel maturity model introduced in Section 4.

FOUNDATIONS
This section introduces the core concepts of maturity models, Machine Learning, and production planning and control processes, as a basis for the remainder of this paper.

Maturity Models
Maturity models serve the purpose of assessing the present status of an application and pinpointing areas for potential enhancement [2].They have emerged as a valuable tool for facilitating the execution of organizational transformation processes, encompassing new projects, within a company, and in a synchronized manner.They assist in designing and implementing transformation projects efficiently, while also providing a comprehensive framework incorporating all required components of the transformation project.Maturity models outline diverse sequences of maturity degrees for various classes of objects, describing anticipated, desired, or typical development paths in discrete rank stages.This approach fosters a holistic procedure and allows for continuous improvement, leading to enhanced performance or quality.Each maturity level is characterized by specific traits and the requisite characteristics for advancement.The specific design fields within each maturity level model play a critical role in pinpointing additional elements that need to be addressed, aiding in further streamlining the complexity of the transformation project.A design field is characterized by a distinct capability area within the transformation project that effectively represents and organizes the subject matter.Additionally, factors for evaluating the single maturity levels of the model are essential components for further segmenting the design fields [3].
The creation or development of maturity models usually involves three principal phases in the design process [3]: 1) Problem Definition: In this initial phase, the focus is on delineating the design scope of the maturity model.This involves identifying the specific need for the maturity model and assessing the maturity of existing areas of application.2) Model Design: Following an iterative approach, the design process unfolds through various steps, culminating in the establishment of the maturity model across distinct yet logically sequential maturity levels.Design areas are specified for examination, with each level linked to these defined areas.Detailed characteristics are then developed or replicated for each design area and its corresponding maturity levels.3) Evaluation: Like any model, a maturity model undergoes evaluation to assess its structural integrity and the efficacy of methods employed in terms of validity, quality, usefulness, reliability, effectiveness, and generalizability.Furthermore, evaluation encompasses ensuring that the model aligns with client requirements and interests, particularly in terms of result provision.
Ultimately, the evaluation aims to determine the extent to which the maturity model fulfills its intended benefits.

Machine Learning
Artificial Intelligence (AI) is a wide discipline of computer science that has been receiving great attention in the recent past.The increasing popularity is due to the success of machine learning technologies and, most recently, large language models, which yield an impact in day-to-day life of people.Success factors for these advancements are (i) the technological developments in the area of compute power, (ii) the availability of large amounts of data, and (iii) progress in research regarding methods and algorithms.
With machine learning (ML) as a sub-discipline of AI, the technologies applied and the tasks to be approached are limited compared to the universal and often exaggerated expectations of general AI.More precisely, ML comprises the technological approach to make a computer program calculate predictions based on a function that is learned from examples.The function that is learned from the training examples is called a (ML) model.Driven by the developments in massively parallel computer hardware, such as GPUs, the specific sub-discipline of deep learning emerged, which is based on artificial neural networks with a large number of neurons and layers, hence often called deep neural networks.
ML can be sub-divided into the fields of supervised learning, unsupervised learning and reinforcement learning.In supervised learning, the training examples are composed of a set of features, also called predictors, and a response, which is a variable dependent on the features.The goal of a supervised learning procedure is to find a function that predicts the response given the features.The learning process can be considered as supervised, since the response is known for the training examples.Once the function is learned, it can be applied to unknown data, i.e., data for which the response is not known, and the prediction will be correct with a certain accuracy.Artificial neural networks (ANNs) is a specific technology to solve supervised learning tasks.In unsupervised learning, there is also a set of features for each training example, but no response variable.Instead of computing predictions, the goal in unsupervised learning is to discover relations between training examples, such as clusters, i.e., subsets of the dataset that might have something in common.Last but not least, reinforcement learning is a sub-discipline of machine learning, where a system being in a given state takes an action to transition into a new state while receiving a reward as a feedback of the action taken.Given this feedback, a reinforcement learning system learns to maximize the rewards, thus learning to take the best action in any given state.
While the theoretical description of any of these ML paradigms is rather generic, it can be applied in a plethora of use cases solving all kinds of difficult problems, where the task can be mapped onto any of the three presented ML formalisms, and a sufficient base of training data of high quality is available.This means on the other hand, that a small data basis or data of bad quality (e.g., biased, flawed, or uncertain data) would lead to unsatisfactory ML models that do not lead to the desired performance in their respective application area.

PPC Process
Production planning and control processes are pivotal in effectively managing production systems, which encompass interconnected components facilitating information and material flow.These systems are typically divided into the management subsystem and the execution subsystem.
The management system serves to task to control and plan production, which involves defining target values to initiate the transformation process within the execution system.Feedback loops are established to ensure continual alignment between actual performance and desired outcomes.Various factors influencing production, such as delivery bottlenecks, machine malfunctions, or unforeseen disruptions, are identified and addressed through proactive measures [4].
Material Requirement Planning (MRP) and its evolved version, Manufacturing Resource Planning (MRP II), serve as vital tools in optimizing resource allocation and scheduling.They enable organizations to efficiently manage inventory, production orders, and capacity planning, thereby enhancing productivity and reducing operational costs.
Furthermore, the production planning and control system, integrated within the broader Enterprise Resource Planning (ERP) framework, orchestrates the seamless coordination of operational resources across different departments.ERP systems encompass multiple modules, including finance and accounting, human resources, and materials management, all interconnected through a centralized database.
This integrated approach not only streamlines organizational tasks but also facilitates data-driven decisionmaking, fostering agility and responsiveness to changing market demands.By optimizing production processes and resource utilization, organizations can enhance their competitiveness and achieve sustained growth in today's dynamic business environment [5].

Factors for Successful Adoption of ML Technologies
IT Infrastructure: A robust IT infrastructure is essential for successful ML implementations, as digitization levels affect ML efficacy.Considerations during planning should encompass scalable high-performance infrastructure to accommodate current and future needs.Incorporating structures like knowledge distillation and Quantum ML can enhance efficiency.
Conversely, non-networked, heterogeneous infrastructures hinder data integration.Transforming legacy IT systems into interconnected networks is crucial for comprehensive ML application [12].

ML Competencies:
Collaboration among multidisciplinary teams, including domain knowledge, fosters ML competency.Foundational ML knowledge is vital for employees and decision-makers to grasp ML's implications.Difficulties emerge due to the shortage of data scientists and personal inclinations.Financial constraints pose challenges for SMEs in attracting ML experts [24].
Machine Learning: Success factors rooted in scientific principles are essential for enterprise ML adoption.ML applications must be tailored to specific industry needs, emphasizing dimension reduction, visualization, and automation.Challenges include inductive biases, black box nature, and overfitting.Keeping pace with ML's rapid advancements poses difficulty in maintaining an overview [25].
Data: Data quality and availability are crucial for ML success.Challenges such as data silos, imbalanced datasets, and data drift hinder ML training.Implementing operational data governance can mitigate these challenges.Approaches like data lakes and federated ML show promise in overcoming data obstacles [26].
Corporate Culture: Management support and corporate strategy influence a company's readiness for ML adoption.A culture of open innovation encourages employee participation and fosters ML application development.Failure to adapt to changing market demands and immature approaches hinder successful ML integration [27].
Project Management: Effective project management is vital for AI application success.Detailed problem definition and use case development are prerequisites.Agile methodologies and regular progress measurements enhance project outcomes.ML implementation requires a thorough understanding of business processes and a collaborative approach.ML implementation presents technical, organizational, and general challenges.Prerequisites for successful implementation include a dissected structure tailored to specific domains and collaborative approaches.Overcoming challenges requires a collaborative effort and a commitment to continuous improvement [28].

ANALYSIS OF EXISTING MATURITY MODELS
Crafting a robust maturity model in the context of PPC requires a thorough review of existing models.Studying and comparing them plays a central role in developing novel maturity models.A particular focus is laid on dissecting design fields and their associated dimensions.This analysis helps identify new relevant design fields to be integrated into a novel model.The analysed models, as outlined in Tab. 1, are categorized based on their connections PPC processes and ML, AI, and data science.Moreover, these models are distinguished by their conceptual origins, as highlighted in Tab. 2. To achieve this, each selected model is briefly described and then their evaluation is illustrated through an example.In order to precisely identify the design fields and their dimensions, an evaluation of the existing models with respect to methodology was conducted.To address gaps in the scientific understanding of classifying and designing maturity models for PPC processes, a deeper analysis regarding the content of selected models was carried out.This entails highlighting and weighing the various emphases of these models to incorporate their unique aspects into a more comprehensive model.Model no. 8, the "ML Operations Maturity Model" from Microsoft [13], was excluded from further analysis due to its unsuitable and unclear structure.Similarly, the "Algorithmic Business Maturity Model" according to Gentsch [7] (model no. 2) and the "Stage-Based Maturity Model for Industry 4.0 & PPC" according to Busch et al. [19] (model no.14) were omitted as they deviate from our specified focus.Model no. 2 overly concentrates on business sectors like marketing and services, while model no.14 primarily addresses Industry 4.0 issues alongside AI and other digital technologies, rather than emphasizing ML within PPC processes.Consequently, there are 11 remaining maturity models that are analysed further with respect to their content.These will be scrutinized according to their designated dimensions.Through this analysis, common topics and aspects shared among these models are identified.These serve as fundamental components (basic reference) for the new model.To avoid unnecessary complexity in evaluation, dimension groups are used to summarize the dimensions of the maturity models under consideration.The overall design fields examined in the evaluation matrix, as illustrated in Tab. 2, were production planning & control, ML system properties, development, training, project management, data, IT infrastructure, corporate culture, expertise and leadership & strategy.The maturity models under examination for further evaluation encompass two primary content domains: one emphasizing the implementation and fusion of data science, AI, or ML, and the other focusing on the modeling of PPC processes.Through comparing these models, it was possible to delve deeper into their distinct content emphases.This comparison revealed that dimensions related to data science, AI, ML, and PPC were selected based on specific practical considerations.However, there were also overlapping dimensions consistently present across multiple models in the analysis.For instance, technological aspects, like those in the realm of ML, were predominantly featured in practice-oriented maturity models.Conversely, IT infrastructure, data, or leadership & strategy, as examples for design fields, were addressed across nearly all examined models.An additional notable aspect is that while several maturity models associated with PPC exist, they do not  incorporate a comprehensive approach to ML.If at all, such approaches typically pertain solely to internal planning and production unit allocation, overlooking the potential for external procurement planning and control.

NOVEL MATURITY MODEL
This section presents the novel maturity model taking into account how PPC processes relate to ML, as well as the maturity classification using the example of the "Data" dimension.

Structure of the New Maturity Model
Thus far, it has been determined that 14 appropriate maturity models have been identified concerning the current subject matter through an assessment of existing maturity models.Through a more thorough analysis, three models were excluded due to their lack of alignment with the research focus, leaving 11 maturity models for further investigation.This underscores the absence of a mature model tailored to the specified research objective: the potential operational integration of ML in PPC, while considering entrepreneurial task areas.The majority of scrutinized maturity models on data science, AI, and ML primarily stem from practical applications.This situation once again highlights the scientific gap in this field, which holds greater significance within the industry.
The new maturity model's structural framework is devised to follow a grid-based format.Within this grid structure, 11 dimensions have been identified and organized into three design fields: organization, technology, and PPC process.In order to ensure clear presentation and enhance practical usability in this article, distinct grid layers are used to particion the individual dimensions of the entire maturity model.Within these layers, the all-embracing design fields are refined, alongside specifying the dimensions to be assessed and the criteria for evaluation.Together, these grid layers form a comprehensive maturity model.A notable benefit of employing this grid-based approach in crafting the maturity model is its flexibility to accommodate the preferences or requirements of the end user.This adaptability allows dimensions to be either omitted or expanded as per user needs.With the provided grid structure, adopting a systematic approach should present no difficulties.The following table offers a summary of the structural framework and allocation of various criteria, derived from the analysis and evaluation of the scrutinized maturity models.
Table 4 Section of the maturity model level for the "Data" dimension For a maturity model to be practically applicable, it must allow for the individual evaluation of design fields and their subordinate dimensions.Thus, the number of maturity levels is determined based on a grid framework, which varies depending on the objective of the task or purpose at hand.The main objective in defining the number of levels should prioritize an approach that facilitates the seamless transition from applied research to practical implementation.In line with this approach, the number of maturity levels for the current model was set at four.A model with more than four maturity levels would not effectively support an approach that is practice-oriented anymore, as it would lack clarity regarding the dimensions described individually.With a subpartition of four levels, the design fields along with their dimensions can be defined explicitly without sacrificing comprehensibility.This allows the user to easily grasp all pertinent information regarding the concerned design fields and assess the current state of the process.Conversely, a too coarse division of maturity levels fails to provide a detailed categorization of the current state, typically indicating only the presence of a deficit without specifying its extent or the sequence of the process.

The Maturity Level of the Dimension
Tab. 3 (assessment matrix for maturity levels) provides an overview of individual design fields, dimensions, and possible criteria.In the following sections, we delve into the respective maturity levels of each dimension, providing a detailed description through the lens of an illustrative example.For a more comprehensive exploration of these maturity levels, readers are directed to Hartl [22].
As mentioned earlier, the dimensions of this maturity model consist of four maturity levels, with each dimension characterized by its specific criteria.The current maturity level can be determined based on these criteria.To put it in simpler terms, one could say that the maturity level increases with the awareness or automation of the interrelated elements in the affected dimensions.A prime example is the dimension "Data", as every ML concept relies on data (see Tab. 4).Specific criteria within this dimension include data availability, data protection, data security, and data quality.In this example, all four maturity levels are described based on these criteria.In the first maturity level, generated data is not yet perceived as a strategic element in the company, and therefore, it is not further categorized concerning availability, protection, security, and quality.Moving to the second maturity level, data is recognized as an essential strategic element, but it is only collected, assessed, and/or shared at the group level or within individual departments.It is only from the third maturity level onwards that data is made available company-wide and assessed based on specific criteria and needs.In the final maturity level (level 4), data is considered an integral part of the company and is accordingly examined, evaluated, and protected, as outlined in the criteria described in Tab. 4. This structured approach has been applied to all 11 dimensions [22].

CONCLUSION
After reviewing literature and evaluating models, it was evident that change processes and related projects should transcend beyond the process level alone.By employing the three specified design fields-organization, technology, and PPC process-we underscore the pivotal role of organizational alignment in strategies, outcome measurement, project management, investment planning, work culture, and more.In this context, projects often face hurdles stemming from inadequate commitment from senior management or insufficient definitions of project goals.The incorporation of new technologies, like Machine Learning, is frequently underestimated or completely overlooked among employees.Similarly, the "Technology" design field emphasizes integrating innovative technologies to secure the company's long-term competitiveness.Particularly noteworthy is the Technology design field's significance in terms of system integration, IT process compatibility, and data-related advantages.
The integration of ML is poised to shape competitiveness across various forthcoming application domains and is increasingly perceived as a disruptive technology by many companies [23].Nonetheless, the current ML research focus leans heavily towards technical aspects like learning algorithms, with relatively scant attention given to the organizational context and success factors tied to ML implementations, and the design principles of ML software solutions.This underscores the need to establish a comprehensive scientific foundation for the practical application of ML.
The aim of the model presented herein is to furnish companies, especially SMEs, with a structured approach to integrate ML technologies into PPC processes.Companyspecific features and solutions must be devised within each company, as these dimensions are context-dependent.Given time and space constraints, not all aspects can be exhaustively covered.There are additional follow-up projects pending in this regard.Further exploration is warranted to evaluate and implement the mentioned dimensions alongside their maturity levels.Moreover, the model can be broadened to encompass additional companyspecific processes through new design fields.Delving into solutions that are rarely used or not at all in current practice, yet are deemed relevant, would also be a promising extension in terms of abstraction, validity, consistency, and benefits.

Table 1
Selected maturity models for further analysis

Table 3
Overview of the maturity level assessment matrix Data assets are not considered strategically relevant & are consequently not stored or utilized, interfaces to extract data are scarce, and real-time data & data exchange are either nonexistent or limited Data protection: Country-specific data protection rights are only considered to the extent legally required Data is disregarded in terms of relevance, completeness, timeliness, and validity Data availability: Data is consciously recognized as a strategic asset & the necessary data sources are increasingly digitized, real-time data & critical data for individual processes are consistently available Analogous to level 3, in addition, data exchange takes place along the entire value chain, realtime data & critical data can be selected & exchanged with partner companies, service providers, etc., as needed Data protection: Analogous to level 3, in addition, further tools such as Federated Machine Learning (Federated ML) are used to generate or optimize ML models while considering the highest privacy guidelines