Management of Time

Definitions

The management of time in streaming systems refers to how streaming computations make use of the explicit and implicit times associated with messages used in the computation.

Background

In a streaming data system, the data consists of messages that enter the system and then undergo various operations such as filtering, transformation, and aggregation. Between processing steps, these messages are transported by message streams such as those provided by Apache Kafka (Apache Software Foundation 2016c). When messages come from disparate sources via different paths, they may be reordered, delayed, or even lost entirely due to propagation delay or hardware or software failures. Processing steps themselves may also delay, reorder, or lose messages due to multi-threading or various kinds of faults in processing.

It is desirable that the processing of these messages be as reliable and timely as possible, but delay, loss, and reordering of messages can make this difficult or even...