Lightweight thread tunnelling in network applications

Abstract

Active Network nodes allow for non-trivial processing of data streams. These complex network applications typically benefit from protection between their components for fault-tolerance or security. However, fine-grained memory protection introduces bottlenecks in communication among components. This paper describes memory protection in Expert, an OS for programmable network elements which re-examines thread tunnelling as a way of allowing these complex applications to be split over multiple protection domains. We argue that previous problems with tunnelling are symptoms of overly general designs, and we demonstrate a minimal domain-crossing primitive which nevertheless achieves the majority of benefits possible from tunnelling.

Introduction

Modern network elements have many software components – for instance to support user-programmability. Software systems in such contexts must address a fundamental concern, that of multiplexing (sharing) the network element amongst many users. Generally, this involves some form of sandboxing to allow code to be executed on behalf of untrusted users [1], [2]. Consequentially the multiplexing scheme must trade off between performance and security as well as other factors.

Sandboxing can be performed either at the language level (by using safe languages such as Java or ML), or at the machine level (by appropriate memory protection and CPU features). There has been much prior work on language-level sandboxing [3], [4], [5], and most current EEs (Execution Environments) rely on these techniques [6], [7]. However, language-level sandboxing lacks flexibility: invocations between components written in different languages must negotiate some common data marshalling format. Language-level sandboxing also reduces the utility of large bodies of pre-existing code by making it harder to re-use them.

Using hardware facilities to control access to memory and schedule the node’s resources is desirable because it allows any language to be used, permitting legacy code re-use. Marshalling can be efficient since native machine formats for data can be used. The main drawback of using hardware to protect the EEs is that communication between protection domains is expensive due to context switches and cache invalidations. Furthermore, these penalties are increasing: as CPU speeds rise, more and more of their performance comes from effective caching of memory contents, branch predictions, and speculative execution. Frequent switching makes these caches ineffective.

Overall, these costs dissuade application designers from placing their modules in separate protection domains, especially if there is a continual stream of data passing through the application.

Thompson wrote: “The UNIX kernel is an I/O multiplexer more than a complete operating system. This is as it should be.” [8]. This vision of a simple I/O multiplexer is one to which we find ourselves drawn once again, this time in the context of Active Network nodes. We introduce Expert, an OS designed specifically for network elements, filling this I/O multiplexer niche [9]. Sample applications include transcoders, protocol boosters [10], and user-supplied routing/forwarding protocols (e.g. multicast variants). In this paper, we describe how Expert’s memory protection architecture and its lightweight thread tunnelling directly support modular hardware-protected applications without suffering an undue performance penalty.

A good multiplexer will schedule the resource it manages. To this end, Expert uses the concept of a path (first introduced in the Scout OS [11]) to represent a flow of packets and their associated processing resources. The alternative of using multiple processes chained into a pipeline causes several problems: (1) extra context switching adds overhead; (2) as each process has its own scheduling parameters, it only takes one under-provisioned process to rate-limit the entire pipeline; (3) per-flow resource reclamation is complex, needing all processes to participate, and atomic revocation may be impossible; and (4) if multiple flows with different service characteristics are to be processed by the pipeline then each process must ensure it sub-schedules internally.

Unlike Scout, Expert paths can seamlessly cross protection domains. As an example, Fig. 1 shows the execution trace of a path which has tunnelled from module A to execute privileged code in module B, which in turned tunnelled into module C before returning back to module B and thence to A. Modules A, B, and C are all in separate protection domains, allowing B and C to implement trusted functionality securely.

Section 2 discusses existing thread tunnelling schemes, and their shortcomings. Section 3 describes Expert’s lightweight thread tunnelling primitive, and how it allows protected modules to be entered. A transcoder used as an example application is described in Section 4, and results quantifying the performance of the tunnelling system and the transcoder built over it are presented in Section 5. Section 6 concludes this paper and suggests areas for further work.