Using visual lifelogs to automatically characterize everyday activities

Abstract

Visual lifelogging is the term used to describe recording our everyday lives using wearable cameras, for applications which are personal to us and do not involve sharing our recorded data. Current applications of visual lifelogging are built around remembrance or searching for specific events from the past. The purpose of the work reported here is to extend this to allow us to characterize and measure the occurrence of everyday activities of the wearer and in so doing to gain insights into the wearer’s everyday behavior. The methods we use are to capture everyday activities using a wearable camera called SenseCam, and to use an algorithm we have developed which indexes lifelog images by the occurrence of basic semantic concepts. We then use data reduction techniques to automatically generate a profile of the wearer’s everyday behavior and activities. Our algorithm has been evaluated on a large set of concepts investigated from 13 users in a user experiment, and for a group of 16 popular everyday activities we achieve an average F-score of 0.90. Our conclusions are that the technique we have presented for unobtrusively and ambiently characterizing everyday behavior and activities across individuals is of sufficient accuracy to be usable in a range of applications.

Introduction

One of the most significant technological developments in our modern world is the development, and deployment, of various types of sensors throughout our society. As a result we can monitor, in an ambient way, many aspects of our lives and our environment. We are particularly interested in wearable sensors which include sensors to directly monitor human behavior as well as sensors in the mobile devices that we can carry around with us.

Lifelogging is the term used to describe the process of automatically, and ambiently, digitally recording our own day-to-day activities for our own personal purposes, using a variety of sensor types. This is opposed to having somebody else record what we are doing and using the logged data for some public or shared purpose. For example, an athlete recording his or her daily training and logging distance, time, etc. would count as a form of lifelog whereas a security firm monitoring a train station to detect anti-social behavior, would not.

One class of personal lifelogging called visual lifelogging is based on using wearable cameras, of which there are several examples now available including SenseCam [14], Looxcie®, GoPro®, Vicon Revue and the recently-announced Memento. These record either still images or video and are taken from a first-person view meaning they reflect the viewpoint that the wearer normally sees and usually they also record data from other wearable sensors. Applications for visual lifelogging are manyfold. Sometimes they are job- or task-related, sometimes we lifelog for leisure, and more recently we’re seeing lifelogging used for health applications as well as applications for real time lifelogging [34].

In this paper we use visual lifelogging in the task of characterizing the activities of the user of a wearable camera, SenseCam. The major contribution of this paper is the proposal of HMM-based models to characterize everyday activities by merging time-varying dynamics of high-level features (concepts). Utilizing concept-concept relationships to map concept vectors to a more compact space by LSA and the extensive experiments on various concept detection performances generated by Monte Carlo simulations are another contribution of this paper, which have not been reported before in semantic analysis of visual lifelogging, to the best of our knowledge. We describe related work in lifelogging followed by an overview of the approaches to managing lifelog data. In Section 3 we introduce the background to our experiments including how we define a vocabulary of 85 base semantic concepts. Section 4 presents a Hidden Markov Model approach to deal with the activities to be detected. We then present our experiments using both ‘clean’ or perfect annotation as well as automatic detection of base semantic concepts, followed by an analysis of two different sampling methods. We conclude the paper with a re-cap on our contribution and directions for future work.