FaaSten Your Decisions: Classification Framework and Technology Review of Function-as-a-Service Platforms

Function-as-a-Service (FaaS) is a cloud service model enabling developers to offload event-driven executable snippets of code. The execution and management of such functions becomes a FaaS provider's responsibility, hereby included their on-demand provisioning and automatic scaling. Key enablers for this cloud service model are FaaS platforms, e.g., AWS Lambda, Microsoft Azure Functions or OpenFaaS. At the same time, the choice of the most appropriate FaaS platform for deploying and running a serverless application is not trivial, as various organizational and technical aspects have to be taken into account. In this work, we present (i) a FaaS platform classification framework derived using a mixed method study and (ii) a systematic technology review of the ten most prominent FaaS platforms, based on the proposed classification framework. Moreover, we present (iii) a FaaS platform selection support system, called \faastener, which helps researchers and practitioners to choose the FaaS platform most suited for their requirements.


Introduction
In the context of cloud computing, the term serverless is typically used to describe a paradigm focusing on cloud architectures that comprise provider-managed components [1]. From the developer's perspective, the decreased control over the infrastructure gives the impression that servers are no longer needed, whereas in fact servers become a provider's burden. One exemplary serverless use case is when different non-managed component types, e.g., Database-as-a-Service and Software-as-a-Service, are combined to implement a Backend-as-a-Service [2].
Function-as-a-Service (FaaS) is a cloud service model enabling the hosting of business logic in a serverless fashion, which makes this model an essential instrument for developing serverless applications. By deploying event-driven, stateless and often short-lived functions to FaaS platforms, developers outsource the maintenance efforts to the corresponding platform provider. As a consequence, functions are automatically scaled without any imposed limits on the amount of new instances. Moreover, in case functions are no longer needed, they are scaled to zero instances, hence eliminating the need to pay for idle application components unlike in other service models that continuously run components, e.g., Platform-as-a-Service [3].
However, in most cases the FaaS programming model is the only common denominator for all available offerings since the underlying technical platform characteristics and supported feature sets vary significantly. For example, one of the main strength of proprietary FaaS platforms lies in the out-of-the-box integration with providerspecific services, which is typically not the case for open source platform offerings. For instance, AWS Lambda can natively be combined with Amazon SQS to use message queues as sources triggering function execution. In contrast, open source platforms such as Kubeless or OpenFaaS provide a more portable way to develop serverless applications. They indeed reduce platform lock-in, favouring more portable serverless applications that are not locked into specific cloud offerings, e.g., by relying on based on Kubernetes [12], a portable application orchestrator, rather than on Amazon's or Microsoft's clouds.
As a result, choosing the most suitable platform becomes a decision problem involving multiple different di-mensions, which affect both the high-level business perspective of project managers and the low-level technical perspective of application developers and operators. For instance, high-level requirements include the license used by a FaaS platform, which heavily influence the choice of the platform when the latter is intended to be incorporated as an internal task-scheduling component for a company's product. Various low-level technical details have to be analyzed too, which might become a serious obstacle for deciding which platform to choose. Examples of such technicalities are available event source integrations, platform-supported tooling, or restrictions on function execution times or on supported language runtimes.
Our objective here is precisely to provide a way to uniformly classify FaaS platforms with the goal of simplifying the decision-making process for specialists with different levels of responsibilities, i.e., managers and technical specialists. To come up with an efficient classification mechanism, we conducted a mixed method study which combines the results from a systematic literature review and the documentation analysis of existing platforms to derive a FaaS Platform Classification Framework, which we present as the first contribution of this paper.
Our classification framework clearly distinguishes between two views: (i) the business view of project managers and (ii) the technical view of developers and operators. These two different views can help organisations in choosing the FaaS platform best suited for their requirements, with the business view helping project managers to first select the subset of FaaS platforms complying with the project and business requirements with a high-level analysis, and without delving into all technicalities of the platforms themselves. Complementarily, the technical view can then be exploited by the technical specialists working on the development and operation of a FaaS-based project to analyze the technical features of FaaS platforms, while at the same time focusing only on those already complying with the project and business requirements.
Based on our classification framework, we conducted a FaaS Platform Technology Review, which constitues the second contribution of this paper. The technology review classifies and compares the ten most popular FaaS Platforms, i.e., the three most used proprietary platforms and the seven highest-ranked open source platforms. Notably, our review provides a first systematic knowledge base that managers and DevOps can exploit to choose the platform best suited for their requirements.
To further support the selection process, we also provide a FaaS Platform Selection Support System, called FaaStener, constituting the third contribution of this paper.
FaaStener is an open source web-based application exploiting the results of our technology review and enabling to search for FaaS platforms through multiattribute queries. This enables, for example, to look for platforms with a certain license and supporting given monitoring and logging solutions.
To summarise, the main contributions of this paper are threefold: (i) A FaaS Platform Classification Framework, which enables characterising FaaS platforms under two different perspectives, i.e., the high-level business view of FaaS project managers, and the low-level technical view of application developers and operators.
(ii) A FaaS Platform Technology Review of the ten most popular FaaS platforms, i.e., the three most used commercial platforms and the seven highest-ranked open source platforms. The review exploits our classification framework to provide a characterisation of the investigated platforms under both the business and the technical perspectives.
(iii) A FaaS Platform Selection Support System in the form of a web-based application called FaaStener, which enables researchers and practitioners to look for the FaaS platforms most suited for the requirements of their serverless projects and to browse through the systematic knowledge base formed by our technology review.
The rest of the paper is organised as follows. Section 2 provides background on FaaS and related technologies. Section 3 illustrates the reseach approach for conducting our study. Section 4 introduces the FaaS Platform Classification Framework. Section 5 presents the FaaS Platform Technology Review, while Section 6 illustrates the FaaS Platform Selection Support System materialising the results of the review. Section 7 and Section 8 discuss potential threats to the validity of our study and related work, respectively. Section 9 finally draws concluding remarks.

Background
The popularity of serverless computing started to rise after Amazon introduced its FaaS platform offering called AWS Lambda [4]. While the term serverless was also used previously in other contexts, it gained most popularity in the context of cloud-native application development [13]. Essentially, serverless applications are composed of components that are managed by third-parties, which significantly reduces control over the infrastructure, thus, minimizing management efforts [2]. Function-as-a-Service is an integral part of the serverless world as it enables hosting business logic in the form of functions that are typically stateless and driven by events. This means that the deployed function code can be triggered by events originating from multiple heterogeneous event sources such as databases, message queues, or streaming platforms. Moreover, functions can be exposed as HTTP endpoints via API Gateways, or invoked on a scheduled basis, e.g., using cron jobs. The actual list of supported event sources and possible ways to integrate events from third party services depends on the employed FaaS platform. Since the provider is responsible for managing functions, autoscaling comes out-of-the-box, also including scaling to zero instances when functions are not needed. This introduces a new, more flexible cost model, where users do not need to pay for idle components as for Platform-as-a-Service deployments. However, in some cases the costs might become less appealing than with classic cloud service models [14]. Moreover, aside from its benefits, FaaS also has some well-known limitations [15] such as the cold start issue, limited execution time which might also vary on different FaaS platforms, or tighter-coupling with provider's specifics due to outsourced management efforts [16]. However, the combination of the above mentioned properties and limitations is what essentially affects the ways how FaaS platforms need to be chosen.
The FaaS technology landscape comprises a variety of heterogeneous platforms, which are offered as-a-service, or can be installed on-premises. For example, Microsoft introduced Azure Functions [5] while IBM started offering its Cloud Functions [7] based on the open source FaaS platform Apache Openwhisk [8]. The landscape of open source FaaS platforms started to evolve rapidly too, also due to the popularity of container orchestration brought by Kubernetes [12]. Multiple open source products such as OpenFaaS [9], Kubeless [10], or Fission [17] that provide serverless, FaaS-based application development experience on top of Kubernetes. Typically, since functions are event-driven, FaaS platforms can be integrated with multiple possible event sources such as databases, messaging and streaming platforms, or provider-specific services such as AWS Alexa [18]. As a result of this heterogeneity, there is an ongoing work on maintaining the list of available technologies with high-level details such as web-site and documentation pointers curated by the Cloud Native Computing Foundation (CNCF) [19]. Moreover, CNCF is also curating the work on standardization of event specification format called CloudEvents [20].

Research Method
In this section, we first recap terminology for core concepts related to the world of serverless and FaaS, which we use throughout this paper. We then proceed by describing the research method's steps.

Terminology
In this section, we define the serverless-and FaaS-related terminology used in the subsequent sections.
Function-as-a-Service Platform / FaaS Platform. We use the term FaaS Platform to describe an environment that enables deploying, managing, and observing function instances, similar to the notion of a FaaS platform used in the CNCF Serverless Landscape [19]. This definition also covers FaaS platforms that require a specific underlying platform, e.g., when a platform must run on top of a container orchestration platform such as Kubernetes, since these kinds of restrictions do not influence the overall purpose of the given product, i.e., to enable the usage of userprovided functions in a serverless fashion.
Event. We base our the definition of an event on the CloudEvents specification by CNCF. More precisely, by using the term event we mean a data record capturing an occurrence of a particular fact in a software system during its operating time [20].
Event Source. As for events, we rely on the definition from CloudEvents specification [20], i.e., an event source is the context where events happen, e.g., a remote database.
Function. We base our definition of a function on the definition from the serverless whitepaper by CNCF [2]. We use the term function to describe an executable snippet of code which can be hosted on the FaaS platform and triggered by events originating from supported event sources, or invoked directly, e.g., via programmatic access provided by available client libraries.
Serverless Application. The term serverless application is used to describe a combination of one or more functions that interact with a set of resources managed by third parties such as cloud providers or other external as-a-service offerings. Essentially, a serverless application represents a logical grouping of resources and functions in particular, which facilitates, e.g., versioning for development and deployment of applications.
Function Orchestrator. To describe a specialized software that enables composing multiple functions by means of workflows, we use the term function orchestrator. This definition is related only to specialized products explicitly tailored for FaaS, e.g., AWS Step Functions [21] or Azure Durable Functions [22], and does not include generalpurpose workflow engines such as Apache ODE.
Function Marketplace. We use the term Function Marketplace to describe dedicated marketplace platforms for offering functions and FaaS-based applications that are distributed by third parties under proprietary or open source licenses for commercial and non-commercial use, e.g., applications that represent a common serverless use case and can be purchased for educational or development purposes. A function marketplace is managed by a marketplace provider, who also defines packaging and description formats of marketplace's products, for instance. A notable example of function marketplace is the AWS Serverless Application Repository (AWS SAR) [23].
Code Samples Repository. A Code Samples Repository is a publicly-available and officially-maintained collection of function and application examples developed for a given FaaS platform, which can be used as a training material or reused for application development. Unlike function marketplaces, code samples are always distributed under open source licenses and typically provided as-is. Technically, it might be a standalone repository or a part of the main platform's repository maintained by the community or the platform producer, e.g., function examples stored in a examples folder with the platform repository. Figure 1 shows the multi-step process we followed based on a combination of academic literature review and documentation analysis. As an initial step we analyzed existing academic publications that focus on criteria-based review of FaaS platforms and we derived an initial FaaS platform classification framework. Afterwards, we selected ten existing FaaS platfoms, we analyzed them, and we refined the initial classification framework based on the newly-discovered data, and used it for the review of these platforms. In the following, we elaborate on the research method's design.

Step 1: Academic Literature Review
To derive a classification framework that covers both academic and industrial views on FaaS platforms, as a first step, we analyzed the existing literature that focuses on reviewing Function-as-a-Service platforms by searching the initial set of publications using well-known electronic research databases, namely ACM Digital Library, arXiv.org, Google Scholar, IEEE Xplore, Springer Link, Science Direct and Wiley Online Library.
Selection criteria. To identify publications relevant to FaaS platforms analysis and comparison, we defined a set of selection criteria. The initial dataset was screened by the authors using adaptive reading depth [24] to identify publications' relevance. We defined the inclusion ( ) and exclusion (×) criteria as follows: Publications that evaluate and compare existing FaaS platforms and function orchestration technologies.
Publications that are written in English.
× Publications not accessible as full-text or not in the form of a full research paper, e.g. extended abstract, presentation, tutorial, PhD research proposal, demo paper, as they do not provide enough details.
As a result, we identified five [25,26,27,28,29] relevant publications after applying the selection criteria to the identified initial set of publications.
Snowballing and Combination. As additional means to identify relevant literature, we applied the snowballing technique [30]. More specifically, we used Google Scholar for forward snowballing, i.e., analyzed all research papers that cite each of the selected publication, and applied closed recursive backward snowballing, i.e., analyzed all research papers cited by each of the selected publication. Afterwards, we applied the selection criteria defined previously in Section 3.2.1, which lead to the identification of six additional publications [31,32,33,34,35,36]. Due to the relatively small amount of relevant publications (11 in total), the combination step was not necessary.

Step 2: Derive Initial Classification Framework
To derive an initial classification framework, we used the keywording technique [37]. The overal process is as follows: (1) at least two researchers separately analyzed each publication from the selected set to identify common concepts and define keywords for them. (2) The resulting keywords were discussed and clustered to form the initial classification framework. At this stage we had a set of generic, high-level categories, e.g., Licensing, Installation, Documentation, Development, Interfaces, and Observability, each coming with a set of concrete criteria. For example, Observability included Logging and Monitoring categories, which listed supported tooling and available integration mechanisms for adding new tools.

Step 3: Search and Select Relevant FaaS Platforms
As a next step, we selected ten relevant general-purpose FaaS platforms for subsequent refinement of our initial classification framework. We defined the following requirements that had to be fulfilled by a FaaS platform to be included for the documentation analysis: A platform must be general-purpose, meaning that it should not tailored for a specific use case (e.g., training AI models).
A platform has to be actively-maintained, i.e., there exist one or more parties consistently contributing towards new platform releases and the code repository is not stale or archived.
We structured the overall process of searching and selecting relevant FaaS platforms using approaches from existing work, e.g., search engine hits analysis [38].
Phase 1: Platforms Search. To select relevant FaaS platforms, we used white and gray literature sources, namely the list of platforms reviewed in the publications chosen during the literature review step described in Section 3.2.1, and the serverless landscape [19] maintained by CNCF that describes the platforms and tooling related to serverless computing. We aggregated the lists of platforms obtained from both sources while checking if the aforementioned inclusion criteria are fulfilled to ensure that we do not miss relevant platforms. For example, we excluded Snafu [39], a research prototype evaluated in one of the publications as it is not a general-purpose and activelymaintained FaaS platform (no commits in the last 2 years). Afterwards, we sorted the remaining platforms by popularity using separate criteria for hosted and installable platforms (described next), and apply the effort bounded stopping criteria [40] by selecting ten platforms in total.  [40], according to which we limited our analysis to ten FaaS platforms, we hence decided to analyse three commercial platforms and seven open source platforms.
To pick the most representative commercial and open source FaaS platforms, we then decided to select the most popular of them. We used different popularity quantifiers for commercial and open source platforms since (i) commercial platforms typically have popularity quantifiers, like quantity of search engine hits or Stackoverflow questions, which are higher with respect to those of open source platforms by orders of magnitude, (ii) such popularity quantifiers can also be misleading for open source platforms, as their name may not be branded and correspond to other projects, and (iii) the developers' interest in open source platforms can be estimated by analyzing publicly available source code-related metrics, while commercial platforms are typically proprietary closed-source products maintained by one party, e.g., AWS Lambda.
We searched for the three most popular commercial FaaS platforms by sorting commercial platforms by Google Search hits, and by using the full platform name as a search query (e.g., "aws lambda", "azure functions", "google cloud functions", and "ibm cloud functions"). As a result, we selected AWS Lambda, Microsoft Azure Functions, and Google Cloud Functions It is worth mentioning that for all three selected platforms the amount of Google Search hits is above 500.000 hits, whereas there is a significant drop in number of hits for the remaining commercial platforms, all of which have less than 150.000 hits.
In contrast, we approximated the overall interest in open source platforms by measuring the amount of GitHub stars given by GitHub users since all open source FaaS platforms are hosted on GitHub [19]. The amount of stars associated with the GitHub repositories of FaaS platforms were extracted using GitHub's API, which resulted in the following list: OpenFaaS, Apache Openwhisk, Nuclio, Fission, Fn, Kubeless, and Knative. This decision was driven by several factors, namely (i) to highlight the trends in open source FaaS platform development community, and (ii) to provide more data for identifying the gaps between open source and commercial platforms, since the latter generally focus less on supporting integration with third parties. Table 1 shows the final list of FaaS platforms selected for the documentation analysis together with the respective documentation sources we considered.

Step 4: Analyze Platforms' Documentation
We analyzed only the documentation provided via official sources such as a dedicated website maintained by the official producer of the platform and its GitHub repository, in case the platform is open source. The documentation was analyzed separately by the authors using the initial classification framework as a baseline. After the analysis, the results were discussed to identify potential refinements for the classification framework.

Step 5: Refine Classification Framework
After the classification framework was modified using the refinements resulting from the analysis in Section 3.2.3, the final version of the classification framework was further analyzed and cross-checked among authors to reduce potential bias (Section 7). The resulting set of concepts was used to derive our FaaS Platform Classification Framework which we discuss in-detail in Section 4.

Step 6: Conduct FaaS Platforms Review
We exploited our classification framework derived in Section 3.2.5 to conduct a technology review of the ten FaaS platforms listed in Table 1. As a first step, each platform was reviewed separately by two authors. Afterwards, the results were verified by the other authors and merged. The conflicting results were discussed and resolved until a unanimous consensus was achieved. The results of the review are presented in Section 5.

FaaS Platform Classification Framework
When choosing the FaaS platform best suited for a software project, various different aspects have to be taken into account, and two different views can be identified. On the one hand, project managers focus on project-and business-oriented aspects when choosing the possible FaaS platforms for shipping the projects they manage, e.g., whether to select a proprietary or open source platform, and its actual licensing. On the other hand, aspects like whether a FaaS platform natively supports certain function triggers or whether it can be integrated with given monitoring and logging solutions are typically not analysed by project managers. Such aspects are rather considered by the technical experts actually developing and operating the project itself.
Following the above idea, we hereafter present our framework for classifying FaaS platforms (derived in Step 4 of Section 3), by clearly distinguishing the business view of project managers from the technical view of developers and operators 1 . Classification categories pertaining to the business view are presented in Section 4.1, while those pertaining to the technical view are presented in Section 4.2.

A Business View for Classifying FaaS Platforms
The business view of our FaaS Platform Classification Framework comprises categories and dimensions of interest for project managers aiming to identify the FaaS platforms complying with the high-level project requirements. These include, for instance, the license under which a FaaS platform is released and whether the platform can be installed on premise or not, as both dimensions can impact on the integration of the platform with the rest of the software developed in a project [3,41].
All dimensions pertaining to the business view of our classification framework are listed and categorised in Figure 2, and explained hereafter.
Licensing. Each FaaS platform is released under some existing licensing, whether open or proprietary. The category Licensing enables classifying platforms under such dimension by allowing to indicate the actual License and the 1 As already explained in Section 3, the classification framework was derived by extracting self-declared information available in FaaS platforms' online documentation. As a result, information that is not self-declared (e.g., vendor lock-in) is not included in our framework. corresponding license Type. Figure 2 reports some possible values for classifying the name and type of the license under which a platform is released, which can anyhow be any existing open source license name and type (such as those recapped by Laurent in his book [41], for instance), or any proprietary licensing option under which a vendor releases its software.
Installation. FaaS platforms currently come in two different forms, as they can be hosted by vendors, installed on-premise, or both. The purpose of the Installation category is precisely to enable classifying FaaS platforms under this dimension by allowing to distinguish their installation Type (i.e., as-a-service, installable). The Installation category also enables to indicate the set of Target Hosts where a FaaS platform can be installed, which is given by a subset of the set formed by existing OSs and cluster orchestrators (some examples of which are in Figure 2). Of course, if a platform only comes as-a-service, then the set of Target Hosts will be empty.  Release. The Release category enables to classify the release Status of FaaS platforms. Possible values for Release Status are listed in Figure 2, following the possible release statuses discussed in the book on software delivery by Humble and Farley [42].
Interface. Existing FaaS platforms offer different ways for interacting with them, and the purpose of the Interface category is precisely to enable classifying platform under this dimension. The Interface category allows to list the supported interface Types, i.e., whether they offer a command line interface (CLI ), an application programming interface (API ), and/or a graphical user interface (gui ). It also enables to classify whether a FaaS platform supports Application Management, i.e., which subset of CRUD operations it offers to manage applications. Finally, the Interface category includes the Platform Administration dimension, which enables indicating whether a FaaS platform offers operations for its own deployment, configuration, enactment, termination and undeployment.
Community. The Community category enables to classify FaaS platforms based on the size, activity, and popularity of their development community. Given that the seven most popular open source FaaS platforms (which drove the development of our classification framework) are all  hosted on GitHub, the obtained Community category explicitly enables to classify platforms based on quantitative information taken from their GitHub repository, i.e., the amount of Stars, Forks, Issues, Commits and Contributors. All such numeric information gives an indication of the size, activity, and popularity of the corresponding repository [43]. Another dimension explicitly included in the Community category is the amount of high-scored platform-related Questions on Stackoverflow , which also indicates the usage and popularity of the platform.
Documentation. The available official documentation for currently existing FaaS platforms is various in comprehensibility and nature. This is the main reason why the business view of our classification framework includes the Documentation category. The latter enables to indicate whether a FaaS platform comes with an official documentation for function development and deployment (with the Functions dimension), i.e., whether it documents how to develop functions that can be executed by the platform, and the actual processes for suitably deploying them on the platform. The Documentation category also enables to indicate whether a FaaS platform officially documents its usage, development, deployment and architecture (with the Platform dimension), i.e., whether it documents the processes to use the platform for running function-based applications, to develop extension to the platform, to deploy the platform, and whether it provides information on the platform's architecture.
Quotas. Finally, some FaaS platforms comes with upperbounds on the size of applications that can be deployed, as well as on the computing resources and execution time that application can get at runtime. The purpose of the Quotas category is precisely to qualitatively indicate whether such upperbounds apply to a given FaaS platform. The Quotas category indeed includes the Deployment dimension, which enables indicating whether the source Code Size and the deployment Package Size are limited or unbounded. It also includes the Runtime dimension, which enables specifying whether CPU, Memory, Storage and Execution Time are limited or unbounded.

A Technical View for Classifying FaaS Platforms
The technical view in our classification framework comprises a set of categories and dimensions that have to be considered by software development and operations specialists willing to identify if a given FaaS platform suffices the low level, technical requirements. For example, a runtime for the programming language used in the company or project might not be supported by the platform which would complicate the development. Figure 3 presents a set of categories helping to classify a given FaaS platform from the technical perspective. In the following, we discuss all categories and their respective dimensions in-detail.
Development. Essentially, a FaaS platform might provide various mechanisms that facilitate the overall function and Development

Runtime Customization supported, not supported IDEs and Text Editors
IntelliJ IDEA, Eclipse, Visual Studio Code, . . .

Version Management
Application  application development experience. Firstly, to start developing functions in a specific programming language, e.g., Python or Java, the platform must support the required Function Runtimes. Note that even in cases when the platform does not support a particular language, it still might be possible to develop function in such language if the platform supports Runtime Customization, e.g., by supplying custom container images as a function runtime. However, since a custom container image has to be created first following the specific set of requirements (e.g., implementing a required interface), instead of simply providing the source code, this option also introduces additional management efforts.
As FaaS platforms typically impose varying requirements on, for example, the way functions have to be implemented [16], the source code development can also be simplified via plugins for popular IDEs and Text Editors such as IntelliJ IDEA [44] or Visual Studio Code [45] that provide support, e.g., for platform-specific syntax highlighting or automated code packaging. Moreover, a plat-form can provide Client Libraries for a set of programming languages that wrap the platform's APIs as a part of platform's Software Development Kit (SDK) to facilitate the development process, e.g., usage of platform-specific libraries for working with typed events' data.
Version Management. An important aspect in FaaS platforms which can simplify several distinct phases of the application lifecycle is Version Management of serverless applications and functions. Apart from facilitating the development process, versioning is also helpful for application deployment, e.g., to support canary releases [46] or bluegreen deployments [47]. It is hence advantageous if a FaaS platform provides mechanisms for managing versions on the level of single Function versions, or entire Application versions, i.e., a combination of functions and resources tracked together. It is worth mentioning that while it is always possible, for instance, to encode version identifiers as part of function or application names or namespaces, however such implicit versioning is less advantageous than the presence of dedicated mechanisms for version management. Additionally, a platform might not support version management on the serverless application level in case no notion of an application is present.
Event Sources. Due to the event-driven nature of FaaS the event sources support plays an important role in deciding whether the platform is suitable for given development requirements. For instance, one of the common FaaS use cases is when a function needs to be exposed as an Endpoint via an API Gateway [3]. Subsequently, several endpoint-related aspects can be considered here, e.g., whether a Synchronous Call or an Asynchronous Call is supported and which protocols can be used for performing such calls, e.g., HTTP or gRPC. Moreover, it might be needed to use a custom endpoint's name instead of the default resource name assigned when exposing the function as an endpoint, i.e., Endpoint Customization must be supported. In addition, in case a secure communication via HTTPS is required, a platform must provide TLS Support.
The next dimension of event sources encapsulates different Data Store types. Since serverless applications comprise non-managed components, we consider only the higher-level storage types [48] such as File Level which includes object stores like AWS S3, and Database Mode that covers SQL and NoSQL databases like Azure CosmosDB.
Another relevant event source is the Scheduler , which allows invoking functions on a scheduled basis. Typically, internally these sources are implemented using cron jobs, hence developers might need to understand the subtle differences in the required format of cron expressions.
Such event source dimensions as Message Queue and Stream Processing Platform are the next important parts for FaaS platforms. Being one of the main integration mechanisms, messaging plays a crucial role in serverless component integration.
Generally, multiple event sources such as AWS Alexa [18], IBM Watson [52], or GitHub [53] are not related to particularly-large dimensions and instead represent particular use cases, which is covered by the Specialpurpose Service dimension. The list of supported specialized service offerings varies drastically from platform to platform and might be a major factor influencing the choice of a suitable platform [16].
Finally, a platform might provide Event Source Integration options, which makes it possible to trigger functions using custom event sources. Examples of integration options might include plugin development or webhook-based integration. Essentially, the integration using proxy components such as event gateways or message queues is possible in the majority of the cases due to the loosely coupled nature of serverless applications. However, the crucial point here is that the platform's documentation describes official ways to integrate custom event sources, for example, company's proprietary applications emitting custom events, which hastens the overall development process.
Function Orchestration. The next significant aspect characterizing FaaS platforms is related to possible ways of orchestrating multiple functions. Apart from connecting functions by means of events and message queues, the specification of workflows [54] incorporating multiple functions is another way to orchestrate complex function interactions. Therefore, support for Function Orchestration brings additional benefits to platform users. In this category, we consider only FaaS-oriented orchestrators such as AWS Step Functions or Azure Durable Functions following the definition of a function orchestrator provided in Section 3.1. Essentially, the majority of function orchestrators are separate offerings that are tailored to work with FaaS platforms and require a separate classification framework. For instance, multiple criteria from the business view can also be used to classify function orchestrators, e.g., installation, licensing, or quotas. Moreover, other orchestration aspects, e.g., expressiveness and extensibility of the underlying workflow language, require a more detailed analysis.
For the sake of brevity, we present the baseline information relevant for developers willing to start defining function orchestrations. Firstly, the Workflow Definition process might vary significantly [26], e.g., a custom Domain-Specific-Language (DSL) can be used to define a function workflow, or even a standard language such as BPEL [55]. Another option is to implement an orchestrating function in a general-purpose programming language such as Java, which will be responsible for calling other involved functions and aggregating the results. Here, the execution of orchestrating functions is controlled by the function orchestrator that handles stateful operations or error handling, for instance, making this option not valid for programming languages that are not explicitly supported by the orchestrator. Another dimension to consider is the presence of documentation for Control Flow Constructs, e.g., to understand how parallel or sequential task execution can be modeled or whether it is possible to handle errors during the workflow execution. Additionally, function orchestrators can impose Quotas on certain aspects of function workflows execution. For example, the overall Execution Time might be restricted or the Task Input and Output Size can be limited to a particular value [16].
Testing and Debugging. Another crucial set of mechanisms is related to testing and debugging of standalone functions and entire serverless applications. While FaaS platforms are typically not responsible for the code development, additional ways to test Functional and Nonfunctional aspects of deployed functions can be provided, e.g., unit and integration testing, load testing, etc. For example, a platform might offer mechanisms for facilitating unit testing with platform-specific libraries, or verifying function invocation using dedicated CLI commands. Moreover, a combination of Local and Remote debugging mechanisms might be provided by the platform. In all these cases, both platform-native and third-party tooling may provide the possible set of solutions.
Observability. The presence of mechanisms for observing serverless applications is a crucial factor for deciding on the FaaS platform. For instance, platforms might provide various Logging and Monitoring options including both platform-native tooling and third-party tooling. Additionally, platforms can provide documented ways to integrate existing tools, e.g., using a push-based or pull-based integration approaches.
Application Delivery. The next category groups together dimensions related to facilitating delivering developed applications. To hasten application deployment, various Deployment Automation tools can be supported by the platform. For example, a platform-native deployment automation tool such as AWS Cloud Formation [56], or a third-party tools such as the Serverless Framework [57] can be present. Moreover, presence of documented CI/CD Pipelining ways can facilitate the DevOps processes.
Code Reuse. The ability to use existing applications as a basis for implementation as well as having access to exemplary code and configuration snippets can facilitate reducing the time to market. Essentially, we distinguish two separate dimensions of Code Reuse, namely the availability of supported Function Marketplaces, and Code Sample Repositories which are actively-maintained by the platform's provider or open source community.
Access Management. Finally, FaaS platforms might provide various Access Management options related to using the platform from both, developer's and user's perspectives. For example, a platform can provide native or support external Authentication mechanisms. Additionally, there might be supported Access Control mechanisms for defining the access rules for functions and related resources that interact with them, e.g., forbidding a function to access a particular data store resources.

FaaS Platform Technology Review
In this section, we present the second contribution of this article, i.e., the results of the FaaS platforms review using the classification framework introduced in Section 4. As for the presentation of our classification framework, we first discuss the review results relevant for the business view, followed by the technical view.

A Business View on FaaS Platforms
The business view of our classification framework (Section 4.1) provides various categories for classifying and comparing existing FaaS platforms at a high-level. These high-level categories can be of help for researchers and practitioners willing to identify FaaS platforms suitable for hosting existing or new FaaS-based applications by focusing only on those platforms that adhere to high-level projects' requirements. Such top-down classification approach helps eliminating the need to delve into technical details of platforms that could be ignored beforehand.
In the following, we present and discuss the classification of the ten FaaS platforms selected in Section 3, based on the categories in the business view of our classification framework presented in Section 4.1.
Licensing. The different licensing options used by the considered FaaS platforms are listed in Table 2. As expected, all commercial solutions are licensed under provider's own proprietary license. Non-commercial solutions instead mainly use the permissive Apache 2.0 License, as easily observable in Figure 4. The only exception is OpenFaaS, which uses the permissive MIT License.  Installation. Table 3 classifies the considered FaaS platforms by indicating whether they are available as-a-service or whether they can be installed on-premises. Not surprisingly, commercial FaaS platforms are all offered as-aservice. MS Azure Functions also supports installing the Azure Functions Host, which enables running serverless applications on Linux, Kubernetes, MacOS and Windows.  All open source solutions are installable on different hosts, with Nuclio also coming as-a-service, i.e., allowing to run FaaS application on hosted instances of Nuclio and providing additional features, e.g., advanced monitoring and logging solutions, and a 24/7 support [59]. The target hosts actually supported by each installable platform are listed in Table 3. Figure 5 instead displays the frequencies of target hosts, i.e., it associates each target host with the amount of platforms that can be installed on it. Kubernetes turns out to be the most supported target host, as all platforms (but Fn) can be installed on Kubernetes hosts. A reason for this is that installable platforms are either developed by extending Kubernetes itself or to be natively integrated with it.

Main Findings: Installation
Most of the open source platforms (6/7) are only available for on-premises installation.
All commercial platforms are offered as-aservice, except some parts of MS Azure Functions that can also be installed on-premises.
Most of installable platforms (5/8) support multiple target hosts, with Kubernetes being the most popular target host among them (7/8).  Table 4 are instead fully open source, and their source code can be found in corresponding GitHub repositories. Notably, Go turns out to be the most employed programming languages for developing the FaaS platforms which code is publicly available on GitHub (including MS Azure Functions), as also clearly visible from Figure 6. All reviewed open source platforms (but Apache Openwhisk) are indeed mainly developed in Go, which emphasizes the relevance of Go for cloud-native development. Additionally, this language choice could indicate that platforms are developed by extending Kubernetes, which is also developed in Go, or with the installation and integration with Kubernetes in mind.

Main Findings: Source Code
The source code of all open source FaaS platforms is hosted on GitHub.
Most of the open source FaaS platforms (6/7) are mainly implemented in Go.

MS Azure Functions is the only commercial
FaaS platform partially open sourcing its components.
Release. The classification of the considered FaaS platforms based on their Release Status is shown in Table 5.
Commercial platforms are obviously all in production. We can also observe that open source platforms show a kind of maturity. Nuclio indeed already comes as a ready-tomarket (rtm) solution, while all other open source platforms (except for Knative) come as stable releases or release candidates, which are the most mature statuses for in-development software. All commercial platforms are production-ready.
Interface. Table 6 classifies considered FaaS platforms under the Interface category of our classification framework, i.e., by showing whether they offer a CLI, API or GUI, and which capabilities are featured through such interfaces. While we can observe that all platforms support all CRUD operations for serverless applications, they considerably vary in terms of interface type and of operations offered to manage the platform. A first distinction occurs between commercial platforms and open source platforms, with the former being the only providing all types of interfaces. In contrast, all open source solutions provide a command-line interface (CLI ) to access and administer the platform, with only 4/7 also providing an HTTP-based API and only 2/7 featuring a GUI. Figure 7 slightly elaborates on Interface Types, by showing their overall frequencies, i.e., by associating each Type of Interface with the amount of FaaS platforms supporting it. It clearly emerges that CLI and API are the most important types to be supported, and this is clearly motivated by need for programming how to automate the management of serverless applications (and of installable platforms, as well). GUI is however also supported by more than a half of reviewed platforms (6/10), as it eases manually interacting with the platform wherever needed. Finally, it is worth noting that a neat distinction between commercial platforms and open source platforms occurs as per what regards the administration of the platform itself. Commercial platforms do not provide any operation to administer the platform itself, which is a logical result * Termination/undeployment can be achieved by stopping/uninstalling the platform instance with host commands.
Only providing guidelines/code of conduct for contributing to the project.
of being offered as-a-service (Table 3). Conversely, given that all open source platforms come as installable solutions (Table 3), they also provide operations for deploying and enacting the platform, with some also providing operations to configure the platform (e.g., to set quotas, as shown in Table 9). Finally, they all neither support nor provide means for terminating or undeploying the platform (e.g., in the form of runnable scripts or binaries). Installable FaaS platforms indeed rely on what available on the target host for being terminated or undeployed, e.g., if deployed with Docker or Kubernetes, they rely on the latters' functionalities to stop and delete the running Docker containers or Kubernetes deployments.

Main Findings: Interface
All FaaS platforms provide a CLI, with most of them (7/10) being also accessible via an API. A GUI is also offered by more than a half of reviewed platforms (6/10).
All platforms support CRUD operations for managing serverless applications.
Open source platforms vary considerably in the way they can be administered.
Community. As an additional indicator of the platform's popularity, in Table 7

Main Findings: Community
OpenFaaS, Apache Openwhisk, and Knative have the highest ratings on GitHub in terms of stars, contributors, and commits, respectively.

Stackoverflow questions show a drastic difference in interest between commercial and open source platforms.
Documentation. Table 8 illustrates the classification of considered FaaS platforms based on the documentation they provide on development and deployment of supported applications, and on the platform itself. All considered platforms widely document how to use the platform itself and how to deploy serverless applications. Most of considered platforms (except for Knative, Nuclio and Open-FaaS) also concretely document how to develop serverless application that can run on the platform, hereby included the available runtime environments and how to exploit the support provided by the platform (see Section 4.2 for further details). In addition, each installable platform documents its deployment, by providing installation guidelines. Notable differences can instead be observed when looking at how considered FaaS platforms document their development and architecture. Apart from the three commercial solutions, which obviously do not publicly document their development or architecture, one would expect all open source solutions to document the architecture of the platform and its development. However, the classification of open source FaaS platforms in Table 8 (for which a visual representation is displayed in Figure 9) clearly shows that this is not the case. Only Apache OpenWhisk, Kubeless and OpenFaaS fully document both their architecture and development. Also Nuclio documents its architecture, but it only provides guidelines to contribute to its development. Finally, Fission, Fn and Knative only provide guidelines or code of conduct for contributing to the platform, without giving information on their architecture.

Main Findings: Documentation
All platforms provide application deployment and platform usage documentation.
More than a half of open source platforms (4/7) do not document their development, with 3/7 also not documenting their architecture.
Quotas. The classification of considered FaaS platforms in Table 9 indicates whether such platforms establish quotas related to various aspect of application's lifecycle, e.g., runtime or deployment. Commercial solutions provide upperbounds to deployable packages' size, and they set timeouts for function executions. In addition, AWS Lambda sets quotas for the memory and storage space available for each function, Google Cloud Functions instead sets quotas for CPU usage and memory, while MS Azure Functions sets an upperbound to the available storage space. All such quotas vary based on the chosen service level, with each as-a-service FaaS platform setting different quotas for different service levels.
Open source, installable FaaS platforms instead mainly come without predefined quotas, with all of them (except for Apache OpenWhisk) being unbounded in all considered dimensions. Fission, Kubeless, Nuclio and OpenFaaS however allow to set quotas on computing resources, so as to configure and manage resource consumption on target hosts. The only open source FaaS Platform coming with pre-defined quotas is Apache OpenWhisk, which natively sets an upperbound to the deployable package and code sizes, and the memory consumption and execution time of running functions. Notably, for all installable FaaS platforms, additional quotas can be set by exploiting the support provided by the target host (e.g., if installing a FaaS platform on Kubernetes, the latter can be configured to set quotas to running deployments).

Main Findings: Quotas
Most of the open source platforms (6/7) do not impose quotas, with 4/7 anyhow allowing to set customised quotas.
Quotas in commercial platforms can be changed by switching to different subscription plans.

A Technical View on FaaS Platforms
As shown in Section 4, FaaS platforms can be classified using numerous categories that comprise heterogeneous sets of criteria related to their technicalities. In the following, we elaborate on the results of reviewing the ten selected FaaS platforms (Table 1) using the technical view in our classification framework.
Development. One of the main technical aspects related to development is whether a platform supports a required function runtime, which eventually allows developing functions in a chosen programming language such as NodeJS or Java. Table 10 shows the details of function runtime support for the list of reviewed platforms sorted alphabetically, and Figure 11 displays the frequencies of supported runtimes, i.e., it associates each programming language with the amount of FaaS platforms supporting the corresponding runtime. The most popular languages are NodeJS and Python support for which is described by 9/10 platforms. Both NodeJS and Python are interpretable languages with a huge community and ecosystem of libraries, making these languages a perfect choice for fast-paced FaaS-based development. The next place is shared by Go, Java, and .NET support for which is stated by eight out of ten reviewed platforms. As shown previously, Go has become an inherent part of the cloud-native development world. Go is a compiled and fast language, which is also used  to implement multiple FaaS platforms, which is an additional reason why Go is supported by most of them. Moreover, both Java and .NET are mature, well-established languages with a large ecosystem of general-purpose libraries making them a good additions to the main list of supported languages. The third and fourth place among supported function runtimes are then occupied by Docker Images and Ruby, respectively. While Ruby is another good example of a popular programming language for web development, a large support for Docker Images worth a special highlight.
Essentially, supporting Docker Images as a function runtime allows developing and running functions in any programming language assuming that all required dependencies can be included together with the function's logic and the resulting container is compatible with the FaaS platform, i.e., a platform is able to invoke the function and pass the event data. However, implementing function as container images requires more effort, since the underlying container has typically to implement a specific interface al-lowing the platform to interact with the function within the container. Docker image can also encapsulate custom binaries, support for which is stated separately by several reviewed platforms, e.g., Nuclio and Kubeless. The main difference is in which kind of artifacts are used as an input for deploying the function, i.e., a Dockerfile or the binary with a specification of its invocation details.
Additionally, supporting Docker images is one of the possible way to enable the Runtime Customization since a function implemented in a non-supported programming language can be invoked by the platform. However, there are other ways to customize function runtimes. For example, in some platforms the runtime support is implemented via dedicated container images, e.g., for running Java8 applications. In such cases, extending the runtime might also mean building a modified runtime container image on top of an existing image, e.g., to add a particular library dependency. Most of the reviewed platforms provide a description of runtime customization options. on the Development category is completed by checking whether each FaaS platform provides plugins for IDEs and rich text editors, as well as language-specific client libraries for programmatic access to the API of the platform. Table 11 shows the details about supported IDEs and Text Editors (e.g., in the form of plugins) and Client Libraries. One can readily observe that the presence of platformspecific IDEs or of plugins for existing IDEs is rarely the case, especially for open source FaaS platforms.

The classification of considered FaaS platforms based
Most of the considered platforms (6/10) instead provide client libraries for existing programming languages, which wrap the platforms' APIs to facilitate the development of serverless applications (e.g., by easing the use of platformspecific libraries to work with typed data or events). Figure 11 displays the frequencies of programming languages for which a client library is provided, i.e., it associates a programming language with the amount of FaaS platforms providing a client library for it (among the six platforms that are known to provide a client library for a programming language - Table 11). The figure again highlights the importance of Go for serverless and cloud-native application development, and it confirm that Go, Java, MS .NET, NodeJS and Python are the most programming languages most supported by reviewed FaaS platforms.

Main Findings: Development
No programming language is supported by all reviewed FaaS platforms.
Go, Java, MS .NET, NodeJS and Python are the most supported programming languages. Docker images are also popular and help using custom programming languages.
IDEs and text editor plugins are mainly offered by commercial platforms.
Most platforms offer client libraries, with open source platforms supporting less languages.
Version Management. The results for versioning support with respect to single functions and application that represent a combination of multiple functions and resources are shown in Table 12. One interesting fact is that some reviewed platforms tend to omit the notion of a serverless application and rather focus on deploying single functions. For example, Knative does not differentiate between functions and serverless applications, since a Service in Knative represents a container. While in theory a service could also contain multiple functions, the documentation does not provide any details on such deployment strategy and how internal functions can be distinguished. As a result, application versioning in such platforms is not really possible even implicitly, e.g., by establishing certain naming conventions for applications. Secondly, most platforms do not offer dedicated versioning mechanisms, which makes only implicit versioning possible. In some cases, there are rudimentary versioning capabilities, which, however, cannot be considered dedicated versioning mechanisms. For example, in Apache Openwhisk there is an internal version property in metadata, which is assigned automatically af-ter the deployment but cannot be managed. The only documentation describing both function-and applicationlevel versioning is AWS Lambda, where applications can be defined and managed using AWS SAM templates.

Main Findings: Versioning
Most open source platforms (6/7) employ implicit versioning of functions, while most commercial platforms (2/3) provide dedicated mechanisms for versioning functions.
The versioning of serverless applications is not explicitly mentioned by 40% of the reviewed FaaS platforms.
Event Sources. The detailed results of the review for this category are shown in Table 13. There are several observations related to the Endpoint event sources category that can be highlighted. Firstly, all platforms provide capabilities to synchronously invoke functions exposed as endpoints using HTTP protocol, whereas the majority (7/10) of reviewed platforms do not document support for asynchronous invocation of endpoints. In addition, almost all platforms (9/10) provide mechanisms to customize endpoint names and the majority of platforms allow securing the endpoint communication using HTTPS.
Scheduled function invocation is also supported by the majority of platforms (9/10), as described by the Scheduler category. The scheduling is typically time-based, and it is implemented by exploiting cron jobs. The actual definition of time-based scheduling however slightly varies among considered FaaS platforms, mainly in terms of the required formatting.
Different considerations instead apply to triggers related to Data Stores and Message Queues. Most open source FaaS platforms (4/7) indeed do not document any support for event sources in the Data Store category, neither for file level data stores nor for relational and NoSQL databases. There are some exceptions: Knative's documentation lists commercial and open source data stores as supported, Apache Openwhisk states support for IBM's NoSQL database called Cloudant, and OpenFaaS described how Min.io 3 . can be integrated. The main reason for this is that open source FaaS platforms come as distinct and standalone products, which typically do not come with a rich set of natively integrated services at a first place, unlike commercial FaaS platforms. Not surprisingly, all commercial FaaS platforms provide built-in support for multiple data stores originating from the same provider, e.g., AWS Lambda can be seamlessly integrated with the object storage service offering called AWS S3, and Amazon's NoSQL database called DynamoDB.
In the Message Queue category, similar as with data stores support, commercial FaaS platforms focus on provider-specific message queues such as AWS SQS and AWS SNS for AWS Lambda or Azure Queue Storage for Azure Functions. As can be seen in Table 13, the support for message queues is stated by the larger part (4/7) of the reviewed FaaS platforms. This list includes both open source messaging solutions such as RabbitMQ, and commercial solutions such as Google Cloud Pub/Sub [61]. Likewise, for the Stream Processing Platform category, commercial FaaS platforms provide support for providerspecific services such as Google Cloud Pub/Sub for Google Cloud Functions or Azure Event Hubs for Azure Functions. In case of open source platforms, Apache Kafka is the most supported (6/7) stream processing platform, whereas in some cases support for commercial stream processing offerings is provided as well.
For the Special-purpose Service category, there is no clear pattern on the choice of supported services. On the one hand, commercial providers focus on integrating their FaaS offerings with provider-specific services such as AWS Cognito or Microsoft Graph. On the other hand, open source platforms support a variety of external services, e.g., GitHub [53], Slack [62], or IFTTT [63]. To a large extent, the availability of certain services is driven by the needs of open source contributors, making platforms very heterogeneous under this dimension.
Finally, as shown in the Event Source Integration category the reviewed platforms mention multiple possible ways to integrate custom event sources, e.g., plugin development, webhooks, or polling. Essentially, one of the most common ways to integrate custom event sources that is not always listed explicitly is by means of messaging. However, since out-of-the-box integration typically requires less effort, it is more beneficial to have an explicitly-supported integration for a desired event source.

Main Findings: Event Sources
All platforms support synchronous, HTTPbased function invocation, whereas asynchronous calls are supported by 3/10 platforms.
More than a half of open source platforms (4/7) do not support data store event sources.
Schedulers and stream processing platforms are supported by most platforms (9/10). Messaging is also widely supported (7/10). Support for special-purpose services varies significantly from platform to platform.
More than a half of reviewed platforms (6/10) support the integration of custom event sources.
Function Orchestration. Table 14 presents the results of the platforms review with respect to the Function Or- Table 13: Classification of considered FaaS Platforms, based on the Event Sources category in the technical view of our classification framework. n/s stays for "not specified", meaning that no related information is provided in the documentation. To be used for testing purposes 3 Feature is in a pre-release state Table 14: Classification of considered FaaS Platforms, based on the Function Orchestration category in the technical view of our classification framework. C denotes custom DSL, and O denotes orchestrating function, with the list of supported programming languages for developing orchestrating functions given in square braces. The abbreviations "n/s" and "n/a" stay for "not specified" and "not applicable", respectively. n/s Google Cloud Functions n/s n/a n/a n/a Knative Knative Eventing C * n/s Kubeless n/s n/a n/a n/a

MS Azure Functions
Azure Durable Functions O [C#, JavaScript] n/s Nuclio n/s n/a n/a n/a OpenFaaS n/s n/a n/a n/a * Only sequence and parallel execution are supported.
chestration category. Orchestrating multiple functions is an important task, with a the majority of reviewed FaaS platforms (6/10) providing a dedicated function orchestrator aimed to facilitate this task. Furthermore, in most cases orchestrators are provided as standalone components or service offerings, e.g., Openwhisk Composer or Azure Durable Functions. While it is also possible to use generalpurpose workflow engines, e.g., Google Composer is a general-purpose workflow engine based on Apache Airflow which can be used to compose functions via generic HTTP operators, in this review we focus on dedicated function orchestrators.
Half of the reviewed orchestrators allows defining workflows using so-called orchestrating functions, which define the required control flow involving multiple functions. With this approach, orchestrators typically provide a smaller set of supported programming languages for defining workflows, in comparison with supported function runtimes. Moreover, a set of supported control flow constructs such as sequences, exclusive and parallel gateways, is not always defined by the language itself. For example, Azure Durable Functions relies entirely on the constructs of the underlying language, i.e., JavaScript or C#, whereas Openwhisk Composer while also allows defining orchestrations using JavaScript-based functions, also introduces a set of so-called combinators, i.e., module-specific commands representing workflow constructs. For instance, a loop in Openwhisk Composer-based workflow is defined via a module-specific command composer.repeat instead of the regular for loop. AWS Lambda, Fission and Knative instead allow orchestrating functions by relying on DSL-based workflow definitions, e.g.,, AWS Step Functions provides a custom DSL for composing functions on AWS Lambda. The list of supported control flow constructs is defined by the DSL itself, and it can vary significantly. For example, the Eventing component of Knative allows defining sequences and parallel executions, while the orchestrators provided by AWS Lambda and Fission feature more expressive power enabling the description of more complex workflows.
Finally, most platforms do not specify any quotas for limiting the execution of the workflows orchestrating functions. The only exceptions are Apache Openwhisk and AWS Step Functions, which allow to set timeouts for establishing maximum task execution time, with AWS Step Functions also allowing to limit I/O size.

Main Findings: Function Orchestration
More than a half of reviewed FaaS platforms (6/10) support function orchestration by providing a dedicated function orchestrator.
50% of offered orchestrators rely on custom DSLs for workflow definitions, and the other 50% exploit orchestrating functions.
All FaaS platforms supporting function orchestration document control flow constructs.
Testing and Debugging. Table 15 illustrates the review results regarding platform-specific testing mechanisms. The majority of reviewed FaaS platforms (6/10) documents some features related to functional testing. As testing of the function code is not a direct responsibility of a FaaS platform, most platforms just provide mechanisms for invoking deployed functions for testing purposes, e.g., using a dedicated CLI command or GUI. In particular, Apache Openwhisk provides a tool for testing the invocation of NodeJS functions, AWS Lambda provides a GUI to invoke deployed functions for testing purposes, and Fn offers a function development kit (FDK) for Java, which facilitates implementing unit tests. AWS Lambda, Google Cloud Functions and MS Azure also provide guidelines on how to exploit external tooling for for functional testing of serverless applications, typically referring to the platformspecific IDE plugins they provide (formerly reported in Table 11). AWS Lambda also illustrate some aspects of non-functional testing, and in particular related to load testing of serverless applications.
The review results related to provided debugging mechanisms are shown in Table 16. One observation is that all reviewed FaaS platforms document mechanisms mainly related to local debugging. Not surprisingly, the commercial platforms provide advanced debugging solutions, as all of them provide dedicated tooling for step-through debugging of functions. Open source platforms instead mainly document log-based debugging, which can be done using either platform-specific tooling or tools supported by the underlying hosting platform, e.g., the kubctl tooling from Kubernetes. Only in few cases (2/10), platforms do not explicitly provide any information related to debugging, even if they may still support log-based debugging or debugging through available external tools. Overall, both testing and debugging mechanisms provided by open source FaaS platforms are rather rudimentary, whereas commercial platforms often offer more features.

Main Findings: Testing and Debugging
The majority of reviewed FaaS platforms (6/10) documents possible options for functional testing. Commercial platforms also provide more advanced options.
Most of reviewed FaaS platforms (8/10) support local debugging of functions.
Open source FaaS platforms mainly support testing throughout function invocations, and log-based debugging.
Observability. This category covers logging and monitoring dimensions of FaaS platforms, and the corresponding classification of considered platforms is shown in Table 17. As with other tooling-related categories, commercial platforms provide standalone solutions for logging and monitoring. For example, Amazon offers AWS CloudWatch [64] and AWS CloudTrail [65] for monitoring and logging of functions hosted on AWS Lambda. Like- Table 15: Classification of testing mechanisms for considered FaaS Platforms. F and N denote functional and non-functional testing mechanisms, respectively. The abbreviation "n/s" stays for "not specified", meaning that no related information is documented.  wise, Microsoft provides Azure Application Insights [66] and Google offers its Operations suite (formerly Stackdriver) [67].
In contrast, open source platforms rely on external tooling, e.g., using a combination of Prometheus [68] and Grafana [69] for monitoring, or combining Elastic-Search [70] and Kibana [71] for logging. Nuclio and Knative also support the integration with commercial monitoring and logging services, i.e., Azure Application Insights and Google Cloud Operations, respectively. Most reviewed open source platforms (5/7) document possible ways to integrate external logging and monitoring tools. Essentially, the integration can be achieved either by configuring the FaaS platform to send events and logs to an external endpoint (push-based) or vice versa, i.e., when an external component pulls events and logs from the platform (pull-based). Additionally, Fission allows enabling Istio, a service mesh, and installing add-ons, e.g., for monitoring and distributed tracing.

Main Findings: Observability
All commercial platforms offer dedicated tooling for the monitoring and logging of functions.
All open source platforms rely on integration of third-party tooling for supporting the monitoring and logging of functions.
Application Delivery. The results of the review related to the aspects of application delivery are shown in Table 18, with several observations that can be made on deployment automation. Firstly, most platforms rely on some form of  [72], either by using proprietary tools and formats (e.g., Azure Resource Manager or AWS SAM) or by relying on the deployment automation featured by their underlying hosting environments (e.g., declarative deployments using custom resource definitions for Kubernetes-based platforms). In open source platforms, the deployment is automated typically by using a platform-native CLI tool that takes a declarative application specification as an input. Another thing worth mentioning is that the Serverless Framework, a popular solution for automating the deployment of FaaS-based applications, is explicitly mentioned only by Kubeless.
Most platforms (6/10) do not provide any information on CI/CD pipelines integration. Moreover, only Open-FaaS describes possible integration with Jenkins, by also providing a CI/CD-optimized version of the platform that already comes with out-of-the-box integration, i.e., Open-FaaS Cloud. Commercial platforms instead typically have a dedicated CI/CD service, i.e., AWS CodePipeline, Cloud Build, or Azure Pipelines.

Main Findings: Access Management
Commercial platforms natively support authentication and resource access control.
Open source platforms typically rely on features offered by the hosting environment to enforce authentication and resource access control.

FaaStener: Platform Selection Support System
To facilitate the storage and usage of collected data for the FaaS selection process, we implemented an open source FaaS Platform Selection Support System, called FaaStener 4 . FaaStener is a web-based application developed in Angular and Angular Material components library 5 providing a graphical user interface for interacting with the platform's data ( Figure 12).
FaaStener enables users to browse the information for a chosen reviewed platform, to perform a multi-attribute search for suitable FaaS platform based on the specified requirements (e.g., taken from the business, technical, or a combination of both views) and to visualize the corresponding results, which significantly improves the usability of our classification framework and technology review. Another advantage is the possibility to perform feature-wise comparison of multiple platforms by simultaneously looking at their classification in separate tabs of the browser. It is finally worth noting that FaaStener has been intentionally designed to ease the maintenance and extension of the systematic knowledge base it enables browsing, which is currently constituted by the results of our technology review. The technology review results are indeed encoded as JSON files, which structure represents the overall structure of our classification framework. Since FaaStener is open sourced, updating information on an already reviewed FaaS platform or adding a new platform just require to update or upload the corresponding JSON files, by exploiting the mechanism of pull requests in GitHub, which allows to accept contributions from all interested parties. To facilitate the contribution process, FaaStener provides the relevant documentation and resource links in a dedicated site section.

Threats to validity
Wohlin et al. [73] provide a standardised classification of threats of validity potentially affecting secondary studies. Four of such potential threats may apply to our study, namely threats to external validity, to construct and internal validities, and to conclusions validity. We hereafter discuss them and illustrates the countermeasure we adopted to mitigate them.
External validity concerns the applicability of a set of results in a more general context [73]. Since we focused on self-declared information available on the official websites and GitHub repositories of FaaS platforms, our results and observations may only be partly applicable to the broader practices and information available on FaaS platforms, hence threatening external validity. To reinforce the validity of our findings, we organised 3 feedback sessions during our analysis of the documentation available on the websites and GitHub repositories of considered FaaS platforms. We analyzed the discussion following-up from each feedback session, and we exploited this qualitative data to fine-tune our classification framework, the actual classification of considered platforms, and teh applicability of our findings. We also made our data easily accessible throughout the FaaStener, which is open source, and whose GitHub repository includes all artifacts we produced during our analysis. We believe that this can help in making our classification framework, technology review and observations more explicit and applicable in practice.
Construct and internal validity instead concern the generalizability of the constructs under studies and the validity of the methods actually exploited to study and analyze data, respectively [73]. These inherently includes the possible biases affecting our study. To mitigate both above threats, we adopted various inter-rater reliability assessment rounds, aimed at avoiding biases by triangulation (Section 3). We indeed performed various iterations among the authors both for (i) refining the initially obtained classification framework and obtaining that show in Section 4, and for (ii) cross-checking the actual classification of considered FaaS platforms until a full agreement among all authors was achieved.
Finally, threats to conclusions validity may apply depending on the degree to which the conclusions of a study are reasonably based on the available data [73]. To mitigate this threat, we exploited theme coding and inter-rater reliability assessment to limit observer and interpretation biases, both while developing the classification framework and while actually classifying the considered FaaS platforms. The ultimate goal in both cases was indeed to perform a sound analysis of the data we retrieved from official websites and GitHub repositories of considered FaaS platforms. In addition, the conclusions drawn in this paper were independently drawn by the authors, and they were then double-checked against the available information in joint discussion sessions.

Related Work
There already exist some research efforts qualitatively classifying and reviewing FaaS platforms. Such efforts were already listed in Section 3, as they were reviewed during Step 1 of our study and exploited in Step 2 to derive the initial version of our classification framework. Throughout the subsequent steps, we extended the set of criteria considered while classifying FaaS platforms, as well as the set of considered platforms themselves.
A concrete example in this direction is the review by Kritikos and Skrzypek [25], who conducted an assessment of serverless frameworks using a set of criteria corresponding to various phases of a serverless application's lifecycle, e.g., design, development, deployment, or execution. In their work, Kritikos and Skrzypek distinguish between provisioning and abstraction frameworks. The former are responsible for provisioning serverless applications by enabling a "mini-serverless platform", whereas the latter abstract away the technical details of two or more serverless platforms. As a result, such Kubernetes-based FaaS platforms as Fission and Kubeless are classified as provisioning frameworks, and the framework Serverless is described as an abstraction framework. The Authors then evaluate and compare both provisioning and abstraction frameworks by considering 14 qualitative criteria. Our effort goes a step further in two directions by (i) considering a wider set of criteria to analyse and classify FaaS platforms, and by (ii) considering a wider set of FaaS platforms.
Other examples of criteria-based reviews of FaaS platforms are those by Lee et al. [28], Lynn et al. [29], and Mohanty et al. [27], Lee et al. [28] evaluate the performance characteristics of production FaaS platforms, and include a feature-wise comparison of the chosen products, i.e., FaaS platforms from Amazon, Microsoft, Google, and IBM. Lynn et al. [29] evaluate seven enterprise serverless platforms with the term serverless being used interchangeably with FaaS. The chosen products are reviewed based on the set of defined criteria and include, e.g., AWS Lambda, Microsoft Azure Functions, and Google Cloud Functions. Mohanty et al. [27] conduct a feature-wise comparison consisting of 15 features together with performance evaluation of four open source FaaS platforms, namely Kubeless, OpenWhisk, Fission, and OpenFaaS. In this work, the Authors put more focus on performance evaluation of open source platforms.
Some works focus on evaluating FaaS-related, complementary concepts such as function orchestrators and serverless application repositories. López et al. [26] evaluate and compare three commercial FaaS orchestrations systems, namely AWS Step Functions, IBM Composer, and Azure Durable Functions. The chosen function orchestrators are reviewed based on the set of defined criteria, and their runtime overhead is evaluated experimentally. Spillner [74] investigates how FaaS-based applications are specified, stored, and offered using the AWS Serverless Applications Repository. Various statistics are presented related to different aspects of functions' lifecycle.
All the aforementioned research efforts, together with other existing criteria-based reviews of FaaS platforms [31,32,33,34,35,36], provide valuable contributions by taking various angles for reviewing FaaS platforms and related relevant concepts. At the same time, the features considered in all such efforts for classifying FaaS platforms, as well as the classified FaaS platforms themselves, do not overlap, hence scattering knowledge across different pieces of literature. We instead aim to provide a more comprehensive classification framework and technical review, by (i) extending the set features already considered in literature for classifying FaaS platforms, (ii) enabling to classify FaaS platforms at two different levels of detail, based on the involved stakeholders, and (iii) providing a more comprehensive technical review for the most popular FaaS platforms based on our classification framework. In addition, we also provide a first selection support system enabling researchers and practitioners to look for the FaaS platforms most suited to their needs.
Finally, it is worth relating our work to several research efforts tackling the problem of FaaS platforms benchmarking. For instance, Wang et al. [75] analyze the performance, resource management, and architectures of the FaaS platforms from Amazon, Microsoft, and Google. Authors implement measurement functions to discover hidden architecture details of these platforms and collect performance-related data. Our work differs from that by Wang et al. [75] since we focus on feature-wise classification and reviewing of FaaS platforms, rather than on their benchmarking. Similar considerations apply to other research efforts benchmarking FaaS platforms [76,77,78,79].

Conclusions & Future Work
With the ultimate goal of supporting researchers and practitioners in classifying existing FaaS platforms and choosing those most suited to their needs, we presented a classification framework, technology review and selection support system for FaaS platforms. Our FaaS Platform Classification Framework enables the characterisation of FaaS platforms under two different perspectives. Such perspectives are clearly shown in the FaaS Platform Technology Review we presented based on such framework, which provides both a business and a technical view on ten existing FaaS Platforms. The FaaS Platform Selection Support System then materialises the results presented in our technology review in the form of a web-based application enabling researchers and practitioners to submit multiattribute queries for searching for FaaS platforms satisfying their needs, and to compare different platforms based on their managerial or technical views.
Apart from enabling a thorough comparison of existing FaaS platforms, our contributions result in the stemming out of various novel research and innovation directions. For instance, we observed a lack of solutions for testing, debugging and versioning FaaS-based project, especially for separately testing, debugging and versioning of FaaSbased applications from the functions used to implement them. Portability is another example of research direction stemming out from our study, as FaaS platforms resulted to be quite heterogeneous in featured triggers, in supported orchestration, monitoring and logging solutions, and in deployment automation technologies. This obviously hampers the migration of a FaaS-based application from a FaaS platform to another, hence requiring solutions for enhancing their portability.
In addition of the above listed directions for future work, we actually plan to extend the results and support provided by the proposed classification framework, technology review and selection support system. We intend to extend the classification framework defined in this work with other dimensions (e.g., security-or performance-related aspects), and to extend the technology review both by reviewing already considered platforms with the novel introduced dimensions and by considering other existing FaaS platforms and/or related tooling (e.g., function orchestrators). In addition, we plan to combine the categories and dimensions forming our classification framework into advanced metrics for evaluating and comparing FaaS platforms, and to exploit such advanced metrics to extend our selection support system into a full-fledged decision support system, which will be capable of providing smart informed recommendations to researchers and practitioners needing a FaaS platform for run their applications.