DeepTSF: Codeless machine learning operations for time series forecasting

This paper presents DeepTSF, a comprehensive machine learning operations (MLOps) framework aiming to innovate time series forecasting through workflow automation and codeless modeling. DeepTSF automates key aspects of the ML lifecycle, making it an ideal tool for data scientists and MLops engineers engaged in machine learning (ML) and deep learning (DL)-based forecasting. DeepTSF empowers users with a robust and user-friendly solution, while it is designed to seamlessly integrate with existing data analysis workflows, providing enhanced productivity and compatibility. The framework offers a front-end user interface (UI) suitable for data scientists, as well as other higher-level stakeholders, enabling comprehensive understanding through insightful visualizations and evaluation metrics. DeepTSF also prioritizes security through identity management and access authorization mechanisms. The application of DeepTSF in real-life use cases of the I-NERGY project has already proven DeepTSF's efficacy in DL-based load forecasting, showcasing its significant added value in the electrical power and energy systems domain.

From an implementation and development perspective, today's software systems are wide open to innovative approaches enabling the incorporation of real-time and data-centric solutions powered by information and communication technologies (ICT) such as big data, internet of things (IoT), artificial intelligence (AI), and lately machine learning operations (MLOps) [5].MLOps is an emerging field in AI that establishes a new paradigm in industrial ML that goes beyond conventional, and manual model development processes.In this context, automated and continuous model training, evaluation, validation, and deployment already replace the manual processes of the conventional ML lifecycle which is the core of current time series forecasting pipelines, leading to faster and more efficient decision making for related stakeholders.
Specifically, Python provides a diverse selection of open-source time series forecasting tools including PyTorch-Forecasting [53], GluonTS [4], Prophet [52], Darts [25], and other relevant options.PyTorch-Forecasting is a Python library built on the PyTorch framework, specifically designed for deep learning-based time series forecasting.It provides a high-level API and supports advanced neural network architectures, enabling users to leverage the power of deep learning for accurate predictions.PyTorch-Forecasting is well-regarded for its flexibility and customization options, making it a preferred choice for researchers and practitioners with expertise in deep learning.GluonTS, built on the Apache MXNet framework [6], is a library focused on probabilistic time series forecasting.It offers a comprehensive suite of tools and models for capturing uncertainty and modeling complex patterns in time series data.GluonTS provides state-of-the-art probabilistic forecasting models such as DeepAR, RNN-based models, and Temporal Convolutional Networks (TCN).Additionally, it offers utilities for data preprocessing, model evaluation, and visualization.Prophet, developed by Facebook, is a widely-used library specifically designed for time series forecasting.It offers a user-friendly interface and employs an additive model that decomposes time series into trend, seasonality, and holiday components.Prophet simplifies the forecasting process by automating trend detection and supporting custom regressors.Its simplicity and ease of use make it accessible to users with varying levels of expertise in time series forecasting.Darts is a versatile library for time series forecasting.It provides a comprehensive collection of models, including classical statistical models, machine learning algorithms, and deep learning architectures.Darts emphasizes automation and MLOps capabilities, offering automated hyperparameter optimization and model selection.
It also seamlessly integrates with other Python libraries, facilitating efficient data preprocessing and feature engineering.
In addition to these tools, there are other notable time series forecasting libraries available in Python.For example, scikit-learn [47], a popular machine learning library, offers various algorithms suitable for time series forecasting along with utilities for model evaluation and selection.TensorFlow Probability [61] combines deep learning with probabilistic modeling, providing a powerful framework for capturing uncertainty in predictions.
In this paper we present DeepTSF, a full-stack machine learning operations (MLOps) framework that provides codeless machine learning (ML) capabilities for time series forecasting by automating several parts of the ML lifecycle.The back-end of DeepTSF is specifically developed to automate the ML and DL-based forecasting lifecycle for data scientists and MLops engineers.It incorporates state-of-the-art ML and DL algorithms and techniques, providing them with a user-friendly and powerful tool for accurate and efficient time series forecasting.DeepTSF harnesses the capabilities of Python's extensive scientific libraries and frameworks, enabling seamless integration with existing data analysis workflows.Additionally, it provides a front-end that can serve several other categories of high-level stakeholders and end-users within the time series modeling domain by providing high-level insights-through visualizations and evaluation metrics-of the forecasting models developed at lower level by the engineers.Ultimately, securing the provided services is a high priority of DeepTSF and is achieved by employing effective identity management and access authorization mechanisms.In terms of real-world application, DeepTSF has already been validated by researchers [49,48] within DL-based short-term electricity load forecasting use cases.However, in practice, DeepTSF can handle all types of forecasting tasks irrespective of the time series domain, resolution, and forecast horizon.

Software description
DeepTSF is a user friendly, AI-based time series forecasting application that facilitates MLOps for multi-horizon time series forecasting with deep learning models.DeepTSF has been developed using various programming tools in Python and Javascript alongside Docker [21] for efficient building, sharing and deployment of the application.

Software Architecture
For the development of DeepTSF, a modular and micro-service-oriented approach has been followed, consisting of several layers that assume different responsibilities and implement various functionalities within the application.In this context, the overall DeepTSF architecture (also illustrated in Fig. 1) consists of several layers that employ state-of-the-art programming frameworks as follows.
• Database: Time series data (e.g.energy and weather-related series data) are periodically stored to a MongoDB [41] (NoSQL database) after several pre-processing and data harmonization processes.MongoDB was selected because it provides great performance and scalability, as it is able of processing millions of requests per day without changes in performance.Furthermore, it provides flexibility to the developers, in the sense that stored objects do not need to follow a strict data model and they map directly to objects in the code.Additionally, it provides the ability to embed related data to a document, to increase performance and reduce computational costs.Of course, there is a wide variety of (big) data storage technologies that serve different requirements.To this end, we aim to abstract the data storage layer, as DeepTSF aims to support more data storage technologies in the future by giving the users the ability to connect their own databases.• Forecasting backend: This component is the core of DeepTSF and is responsible for: i) extracting data from the database, ii) preprocessing and harmonization in accordance with the specifications of the forecasting model input, iii) model training and (optionally) hyperparameter tuning, iv) model evaluation and v) model storage, versioning and serving.These steps can be executed either sequentially or individually.The execution/orchestration of the pipeline can take place either through CLI or the workflow orchestrator that is described later on.The backend's code is organized in files that correspond to the stages of the above workflow: -load_raw_data.py:This interface is responsible for loading the data from local or online sources and saving it to the MLflow tracking server.The DeepTSF documentation [20] includes more information on the backend's code structure and usage.The most important technologies of the forecasting backend are the following: -Python MLflow [5] as the core of this implementation.The model training client's code extensively uses the MLflow API to allow for machine learning experiment triggering and tracking on the tracking server.Optionally, models can also be registered, versioned, and deployed using the MLflow model registry [40].
The tracking server is also assisted by a MinIO [38] artifact store for storing models, datasets, and results alongside a PostgreSQL [51] database for storing experiment parameters and evaluation metrics.Note here that a separate docker deployment [32], that was developed in parallel with DeepTSF, is required for the optional set up of the MLflow tracking server.-Python Darts [46] for automated and optimized time series forecasting based on PyTorch.
-Python FastAPI [33] for exposing the information and functionalities of the back-end application as an API to the outside world.Further details on the architectural elements of this subcomponent are provided in B.
• Graphical user interface which is the front-end environment of the DeepTSF is developed using React [7], an open-source, widely-used and very well-documented JavaScript library, which offers versatility and unparalleled efficiency for building user interfaces.Its main advantages lie in the component-based architecture, which enables the developers to create reusable and modular UI components, resulting in faster rendering and optimal overall application performance.Moreover, React offers a robust ecosystem of libraries and tools, ensuring that the created applications will be scalable and maintainable, while there is also a very active community of developers that offers support and contributions to the React library.• Workflow Management DeepTSF supports two layers of workflow management, as follows.
-Basic workflow management The first layer comprises a CLI which has been developed with the Click package [17] and allows for executing (individually or sequentially) the four stages of the ML pipeline that were above mentioned.This CLI is complemented by the built-in CLI of MLflow [39].-Advanced workflow management The advanced workflow management engine is based on Dagster [18].This engine allows for orchestrating, scheduling and executing the ML pipeline in a more advanced fashion.This component is integrated with the MongoDB to extract the data needed for the forecasting scenario, as well as with the MLflow platform for tracking the experiments and storing the trained models.This integration is achieved utilizing the python client APIs of those tools.In addition, the GraphQL API of Dagster [24] is utilized to expose functionality of executing the defined jobs to the end-users of application.Note here that every forecasting use case requires a new custom configuration of the Dagster orchestrator, which is abstracted within our work so that it can be used as a starting point for future extensions.• Identity and access management Identity and access management in DeepTSF are implemented leveraging OpenID Connect Protocol and therefore the OAuth2.0protocol on top of which OpenID Connect is built [22].A Keycloak [15] server is used and configured to act as the OpenId Provider.User authentication is achieved via the front-end application, where users authenticate against the OpenID Provider using their credentials.After successful authentication, a token exchange request takes place, resulting to access token issuance and its exchange.Concerning the back-end service of the application, the access to its FastAPI endpoints is validated against OpenID Provider by utilizing its introspection endpoint providing the access token sent within user's request and by role validation logic implemented in FastAPI.Both front-end and back-end services are registered within the OpenID Provider as confidential clients in order to establish a trusted relationship.A reverse proxy setup has been developed to enable authentication for the MLflow server.This reverse proxy service is an OpenResty [44] webservice which extends the NGINX [54] with Lua and act as the OpenID Connect Relying Party authenticating the users against the OpenID Provider.

User interface
The front-end of DeepTSF is designed to provide a smooth experience to the end-user roles within the application (system administrators, data scientists and domain experts).The following sub-sections describe the main functionalities of the DeepTSF front-end environment.More precisely, each page of the front-end application will be presented, along with a detailed description of the inputs and the outputs, as well as the overall functionalities it offers.

Sign-In and homepage
The Sign-In page has been developed to provide the capability of logging into the dashboard, as illustrated in Fig. 2a.After signing in, the homepage displays the main capabilities of the dashboard in head titles, as illustrated in Fig. 2b.Visually, the homepage is separated in horizontal sections, one for each one of the UI's main functionalities.Codeless forecast This page enables the execution of an experiment, after uploading the desired dataset in csv format and configuring key parameters of the model pre-processing, training and model evaluation steps.The provided interface has been divided in small, distinct sections, which guide the data scientist through the needed steps to sufficiently provide the required input for the execution of an experiment as illustrated in Fig. 3.In the Dataset Configuration section, the users can either upload their own files or choose among the already stored ones, before selecting the time series resolution and the dataset split that contains the start and end dates, which represent the dates defining the experiment execution.In the Model Training Setup section, the users can define the experiment name, choose between the available algorithms and choose between a set of hyperparameters that vary depending on the chosen algorithm.Finally, the users have to choose a forecast horizon for backtesting the model during evaluation.After filling all the required fields, the "EXECUTE" button becomes available.Upon clicking the button, an experiment with the entered configuration runs and the user is given the choice to navigate to the MLflow tracking UI to retrieve details about the execution.Experiment tracking / evaluation This page offers the users the capability of evaluating their experiments' results, by either their name or their id as illustrated in Fig. 4.After specifying the main evaluation metric and a number of evaluation samples, the users can press the "DETAILS ON MLFLOW" button, which navigates them to the MLflow instance, where they can access a number of comprehensive experiment details.Moreover, upon clicking the "LOAD METRICS", two charts are displayed on the bottom of the page, as demonstrated in Fig. 5.The first chart presents the model evaluation metrics of the specified experiment, while the second one offers a visual comparison of the actual and the forecasted load series.

System monitoring
The system monitoring page presents a user interface that showcases real-time data pertaining to the overall memory, GPU, and CPU utilization of the deployed infrastructure, as illustrated in Fig. 6.The displayed data are continually refreshed at a one-second interval.To prevent overloading the backend responsible for providing this information, the live demonstration is limited to one minute (60 reloads).This configuration ensures that extended periods of inactivity, such as leaving the tab open, do not strain the backend.As a result, each section includes a "Refresh Live Feed" button, enabling users to re-engage with the live monitoring if desired.

Forecasting interface
As above mentioned, the main ML workflow comprises 4 consecutive steps, as illustrated in Fig. 7.The executed experiments are logged to the MLflow tracking server and can be visualized by the user using the MLflow UI that is demonstrated in Section 3.1.The functionalities of each stage are described in more detail in the following subsections.Data loading DeepTSF can handle univariate, multivariate, and multiple time series, optionally including external variables (covariates).Firstly, the dataset is loaded from local or online sources.Currently Deep-TSF supports csv files of the schema that is included in C. In this context, the connectors that enable data ingestion vary depending on the use case and the schema of the respective data source and shall be engineered by the DeepTSF user.We provide an example of this connector, which works with MongoDB.After that, validation is performed to ensure that the files provided by the user respect the required schema.The files are saved on the MLflow tracking server so that they are available for the data pre-processing stage.
Data pre-processing Then, data pre-processing is performed.Specifically, for each component of each time series, outlier detection is optionally conducted by removing values that differ more than an arbitrary number (defined by the user) of standard deviations from their monthly average, or that are zero in the case of a non-negative time series.Outliers are replaced by missing values.Subsequently, missing data may be imputed by using a weighted average of historical data and linear interpolation as proposed by Peppanen et al. [50].
Training and validation After the pre-processing stage, the data is scaled using min-max scaling, and is split into training, validation, and testing data sets.Then, the training of the model begins using only the training data set.The currently supported models are N-BEATS [45], Transformer [64], NHiTS [16], temporal convolutional networks [9], (block) recurrent neural networks [26], temporal fusion transformers [35], LightGBM [30], random forest [14], and seasonal naive as from the documentation of Darts forecasting models [19].The latter can serve as an effective baseline depending on the seasonality of the time series [49].Hyperparameter optimization can be also triggered using the Optuna library.DeepTSF supports both exhaustive and Tree-Structured Parzen Estimator-based [12] hyperparameter search.The first method tests all possible combinations of the tested hyperparameters, while the second one uses probabilistic methods to explore the combinations that result to optimal values of the user-defined loss function.Ultimately, a method based on functional analysis of variance (fANOVA) and random forests, as proposed by [27] is used to calculate the importance of each hyperparameter during optimization.In this context, a bar graph showing the aforementioned results is produced and stored as an artifact to MLflow.
Evaluation and explanation When model training is complete, evaluation is performed through backtesting on the testing data set.Specifically, for each time series given to the function, it consecutively forecasts time series blocks of length equal to the forecast horizon of the model from the beginning until the end of the test set.This operation takes place by default with a stride equal to forecast horizon but can be changed by the user.Then, evaluation metrics are calculated using the resulting forecasted time series.The evaluation metrics that are supported are: mean absolute error (MAE), root mean squared error (RMSE), min-max and mean normalized mean squared error (NRMSE), mean absolute percentage error (MAPE), standardized mean absolute percentage error (sMAPE), and mean absolute scaled error (MASE) [28].In the case of multiple time series, it is possible for all evaluation sub-series to be tested leading to an average value for each one of the metrics.In this case, DeepTSF stores the results for all time series.Additionally, it is possible to analyze the output of DL and DL models using SHapley Additive exPlanations [36].Each SHAP coefficient indicates how much the output of the model changes, given the current value of the corresponding feature.In DeepTSF's implementation, the lags after the start of each sample are considered as the features of each model.Following that, a beeswarm plot [57] is produced.In addition, a minimal bar graph is produced showing the average of the absolute value of the SHAP coefficients for each attribute.Finally, three force plot charts are produced, showing the exact value of its SHAP coefficients for a random sample.The above mentioned artifacts are accessible through the MLflow tracking UI.

Workflow management interface
As already mentioned in Section 2.1, the workflow management interface comprises two subcomponents: i) a CLI for triggering the stages of the pipeline, ii) a workflow orchestrator based on Dagster that takes care of model re-training and error handling issues.
CLI The DeepTSF CLI is a direct way of handling and triggering the workflow stages easily, automating the argument passing process and linking the execution of each script in a sequential order, passing the proper arguments from one to another.Figure 7 illustrates this concept.More details on the DeepTSF CLI functionality for either the manual execution of pipeline stages or their automated sequential execution can be sought in Section 3 as well as the MLproject file that is contained in A.
Dagster Dagster [18] acts as a workflow orchestration engine, where data processing and ML model training pipelines, are defined as jobs.Therefore, the execution of these jobs can be scheduled in fixed intervals, serving the needs of periodic training.This component interacts with the data source, extracting the data needed for the pipelines, as well as with the model registry, where the models are stored when training is completed.More specifically, the defined jobs in Dagster are in line with the flow described in section 2.2.2, starting with a Dagster asset responsible to load the raw data, from database using the defined resource modeling the MongoDB.The next step of the workflow consists of the pre-processing task, producing the asset that will be consumed by the following training step and (optionally) validation step.When training is complete, the produced trained model is stored to the MLflow Models registry.The last step consists of the model evaluation process.The MLflow tracking server is utilized along steps to track the ML experiment, storing information such as the calculated metrics in evaluation phase, the best parameters computed in hyperparameter tuning, and the configuration options used for the execution of pipeline.The execution of this pipeline is scheduled in order to be in sync with the periodic update of the MongoDB loading the new smart meters data.Ultimately, Dagster provides a web user interface including, among others, information regarding the defined jobs and their runs, the defined schedules, the produced assets, as well as providing the ability to configure and execute these jobs.

User roles
Access to components of DeepTSF is restricted to authorized users.Users should first be registered using the identity and access control mechanism of DeepTSF, in order to use the components of the application.Then, based on their roles, they acquire different levels of access to the application.Supported roles include data scientists, domain experts, and administrators.The "domain expert user role" has limited access to application features that are the "experiment tracking" and "system monitoring" interfaces.The "data scientist" user role has additional access to the "load forecast" feature of the main page while they are also allowed to straight access to the API of the application back-end.The MLflow experiments are available to all users.Lastly, the administrator role have access to all the features of the application.

Illustrative Examples
In this section we provide illustrative examples of how a data scientist can interact with DeepTSF.As already mentioned, there are two ways in which this user can interact with the application: i) through the DeepTSF UI, ii) through the DeepTSF CLI.

A use case on DeepTSF's CLI functionality
In this section we provide an illustrative use case of DeepTSF's CLI functionality which is aimed for coding experts such as data scientists and machine learning engineers.Specifically, we will utilize the CLI for day-ahead short-term load forecasting (STLF) on Italy's national electricity load.Within our example, the N-BEATS deep learning model has been chosen, given its high performance on similar use cases [48,49].
The command that should be given through the terminal to DeepTSF's CLI is shown in the code snippet of Fig. 8.It stores its results in the MLflow experiment named "example", and executes the entire workflow.
• Data loading options: Whether to take the data set from MongoDB or not (from_mongo = false), which data set file to use (series_csv = Italy.csv),whether the data set has dates with the day appearing before the month or not (day_first = false), whether to get the dataset from mongo database or not (from_mongo = false), and if the time series is multiple (multiple = false).• Data pre-processing options: If we want to convert to local timezone (convert_to_local_tz = false), whether to perform imputation with the method described in 2.2.2 or just using linear interpolation (in this case we use the former, linear_interpolation = false), the resolution of the data set (in this case 60 minutes, resolution = 60), if the outliers will be removed (remove_outliers = true), the country code to use for obtaining the holidays for our imputation method (country = IT), and the years of the time series to use (year_range = 2015-2022).
• Training and validation options: How the data will be split in train-validation-test sets (in this case we use years 2015-2019 as a train set, 2020 as a validation set, and 2021 as a test set.To do that we set cut_date_val = 20200101, cut_date_test = 20210101, and test_end_date = 20211231), whether to scale the data set or not (scale = true), which model type to use (darts_model = NBEATS), from which entry point to get the model parameters (hyperparams_entrypoint = NBEATS_example, see D about the way we provide the parameters to DeepTSF), which metric to use as an objective function (loss_function = MAPE) throughout the TPE optimization process, whether we desire to perform hyperparameter optimization (opt_test = true), whether to perform an exhaustive search or use the TPE method (in this case the latter, grid_search = false), the number of trials of hyperparameter search to execute (n_trials = 100), whether to train the model using the GPU, or just the cpu(device = gpu), and whether to ignore previous stages of the workflow that have been previously executed with the same arguments (ignore_previous_runs = t) or use them as a starting point instead.• Evaluation and explanation options: The forecast horizon to be used during backtesting (in this case 24 hours, forecast_horizon = 24), the forecast horizon of the naive method used in MASE (m_mase = 24), and whether to perform an analysis using SHAP or not (analyze_with_shap = False).
All other options not present in the command take their default values.
After the command is run, the pipeline is executed.The various stages appear as separate MLflow child runs of the main run in the MLflow UI (Fig. 9).The user can visualize the results related to each stage by clicking on the corresponding run.The hyperparameter optimization stage's page is illustrated in Fig. 10.The hyperparameter optimization stage was executed for 100 trials, and the best model was saved alongside its parameters.The MAPE objective function was calculated on the validation set (2020).The MAPE value of each trial can be seen in Fig. 12.More detailed results about each trial can be found in the product csv file "NBEATS_example.csv",a sample of which is provided in Table 1.Each trial's hyperparameters, as well as all the corresponding evaluation metrics are included there.

Codeless model training for data scientists via DeepTSF's UI
For a data scientist that desires a fully codeless experience, DeepTSF offers the capability of training and evaluating models through the UI as already mentioned.To this end, the user needs to follow the steps below1 : 1. Use the Sign-In page and provide their credentials (see Fig. 2a) 2. Navigate through the homepage, open the side dashboard and select the "Load Forecast" option (see Fig. 2b) 3. Follow the process described within the "Codeless forecast" paragraph of Section 2.2.1 (see Fig. 3) 4. To evaluate the model in detail, the MLflow UI can be visited as described in the previous section.This can be done by clicking the "Visit MLflow server" button which becomes available after the execution.5. To graphically evaluate the model at a higher-level, the user needs to follow the process described within the "Experiment tracking and evaluation" paragraph of Section 2.2.1 (see Fig. 4 and Fig. 5)

Impact
Since DeepTSF is open-source, it can serve as a reference production tool for stakeholders such as data scientists, and domain experts that need to develop, optimize, evaluate, serve and monitor ML and DL models for time series forecasting.DeepTSF makes MLOps for time series forecasting easy for everyone by unifying the ML lifecycle development and monitoring.This is achieved through a user interface that guarantees a user-friendly and codeless experience which is its main contribution to the industrial time series forecasting domain.Additionally, CLI capabilities are also available for data experts and ML engineers that require custom model training procedures and flexibility, given a specific degree of coding expertise.
Thanks to its design, DeepTSF enables the codeless development, initialization, and seamless monitoring of ML experiments.In this fashion, the mass execution of ML experiments is facilitated.Therefore, researchers can easily reproduce and evaluate an abundance of research results related to various ML and DL models and hyperaparameter setups.In this fashion, DeepTSF reinforces the ability for research contributions within the time series forecasting domain.Within industrial environments, DeepTSF can accelerate the processes for the deployment of accurate time series forecasting models in production as domain experts are allowed to directly monitor the model development process via intuitive user interfaces that match their domain knowledge even if they possess limited modeling skills.
DeepTSF is already used in the EU H2020 research project I-NERGY [29] and it was developed as a solution to enable automated energy time series forecasting for both generation and consumption time series for multiple pilot sites and use cases within the project.The service was meant to serve both data scientists coming from the technical partners of the project and energy experts coming from the pilot partners and coordinate their every day time series forecasting tasks in an efficient and highly collaborative fashion.
5 Conclusions and future work

Conclusions
In this paper, we introduced DeepTSF, a comprehensive machine learning operations (MLOps) framework designed to revolutionize time series forecasting with codeless machine learning (ML) capabilities.DeepTSF automates several aspects of the ML lifecycle, making it an ideal tool for data scientists and MLops engineers engaged in ML and DL-based forecasting.
By incorporating cutting-edge ML and DL algorithms, DeepTSF empowers users with a robust and user-friendly solution for precise and efficient time series forecasting.Leveraging the power of Python's extensive scientific libraries and frameworks, DeepTSF seamlessly integrates into existing data analysis workflows, enhancing productivity and compatibility.DeepTSF also includes a front-end that caters to various high-level stakeholders and end-users in the time series modeling domain.Through insightful visualizations and evaluation metrics, these stakeholders gain a comprehensive understanding of the forecasting models developed by engineers at a lower level.Security is also top priority for DeepTSF, and it ensures protection by implementing effective identity management and access authorization.
DeepTSF is versatile and capable of handling all types of forecasting tasks.In real-world applications, currently DeepTSF has already proven its efficacy in DL-based short-term electricity load forecasting use cases with significant added value in the electrical power and energy systems sector.

Future work
With respect to future work we plan to further extend DeepTSF's workflow orchestration interface so that it can seamlessly handle the data and ML pipelines through DAGs, therefore fully replacing the capabilities that are currently supported by the CLI.
Additionally, regarding model serving, it is envisaged that DeepTSF will also integrate with other model serving mechanisms such as Seldon [55], BentoML [11] and interoperability frameworks such as ONNX [43], as MLflow's model serving features are currently on experimental stages.
Last but not least, it is of crucial importance to make DeepTSF solution agnostic in terms of database technology, and data formatting standards, aiming to provide flexibility with respect to the integration with other user-preferred technology stacks and requirements.In this context, the data ingestion mechanism of DeepTSF is planned to be extended to support the integration with various data sources and database technologies, primarily focusing on time series databases such as InfluxDB [2] and Timescale [62].Furthermore, interoperability frameworks will be utilized to facilitate different data schema standards.

Current code version
Table 2 provides information about the current code github repository, documentation, and versions.

A MLproject file
The MLproject file describes the way the ML pipeline is executed.It is in YAML format, and it defines the name of the MLflow project (in this case DeepTSF_workflow), the file to use to build the environment the user desires to work with (in this case conda.yaml),and the entry points of our project.
Each entry point corresponds to a specific stage of the pipeline describing the respective python command alongside its parameters.In this case, each entry point runs the main python file for the stage it corresponds to, and passes the parameters to the file using the Click library [17].The file is shown in the next pages:

C Permitted file format
The format of the csv files DeepTSF can accept depends on the nature of the problem it is trying to solve.More specifically, in case of a single time series file, its format is illustrated in Table 3: The columns that can be present in the csv have the following meaning: • Index: Simply a monotonic integer range • Date: The Date each row is referring to • ID: Each ID corresponds to a component of a time series in the file.This ID must be unique for each time series component in the file.If referring to country loads it can be the country code.In this case, this will be used to obtain the country holidays for the imputation function as well as the time covariates.• Timeseries ID (Optional): Timeseries ID column is not compulsory, and shows the time series to which each component belongs.If Timeseries ID is not present, it is assumed that each component represents one separate series (the column is set to ID). • Time columns: Columns that store the Value of each component.They must be consecutive and separated by resolution minutes.They should start at 00:00:00, and end at 24:00:00 -resolution The checks that are performed when valifating a file are the following: For all time series: • The dataframe can not be empty • All the dates must be sorted For non-multiple time series: • Column Datetime must be used as an index • If the time series is the main dataset, Load must be the only other column in the dataframe • If the time series is a covariates time series, there must be only one column in the dataframe named arbitrarily For multiple timeseries: • Columns Date, ID, and the time columns exist in any order • Only the permitted column names exist in the dataframe (see Multiple timeseries file format bellow) • All timeseries in the dataframe have the same number of components For more information about these files see the documentation [58].

D Providing the hyperparameters to DeepTSF
To use DeepTSF's optimization mechanism the user needs to provide the desired hyperparameter grid in the con-fig_opt.ymlfile using the YAML format [65], as shown in Fig. 16.This is the grid used for the example of 3.1.In the YAML file, the possible values for each hyperparameter need to be given in a list format as follows: • Format ["range", start, end, step]: a list of hyperparameter values are considered ranging from value "start" till "end" with the step being defined by the last value of the list.• Format ["list", value_1, ..., value_n]: All the listed parameters ({value_1, ..., value_n}) are considered in the grid.
-etl.py: Conducts data harmonization and pre-processing.-training.py:Performs model training.-optuna_search.py:Responsible for the hyperparameter tuning of the model if requested by the user.-evaluate_forecasts.py:It evaluates the model using the provided test set.

Figure 2 :
Introductory web pages of DeepTSF

Figure 5 :
DeepTSF experiment tracking and evaluation page results.

Figure 8 :
Figure 8: Full command given to the DeepTSF CLI for the execution of the Italian STLF use case

Figure 11 :
Figure 11: Time series line plot after the imputation of missing values by DeepTSF

Figure 12 :
Figure 12: The values of the objective function (MAPE) for each hyperparameter optimization trial

Finally, the model
is evaluated at the evaluation stage.The model's prediction of the test set, along with the actual values of the series can be visualized in Fig.13.The resulting values on the test set of all the metrics DeepTSF supports are saved as MLflow metrics (see Fig.14).

Table 1 :
Detailed results of all hyperparameter optimization trials

Table 3 :
Single time series file format in hourly resolutionIn this table, the Datetime column simply stores the dates and times of each observation, and the Value column stores the value that has been observed.If we are solving a multiple and / or multivariate time series problem, then the file format (along with example values) is shown in Table4:

Table 4 :
Multiple and / or multivariate time series file format