From Sensorimotor Features to Knowledge and Ontology for Scene Analysis in AAL

Scene analysis and activity recognition in an AAL context are both challenging problems because they often require systems that are able to perceive and act in highly unstructured environments. Not only are these environments dynamic, they usually also consist of multiple agents acting according to their own beliefs and intentions. Biological cognitive systems are still superior to technical systems regarding their adaptivity and robustness in such non-restricted real life situations. During evolution they developed optimal solutions for the use of multisensory information, for exploiting information on different levels of granularity, and for reasoning and acting with this information in a multitude of different situations that are never really identical. We present a number of our own applications that illustrate a range of complexity found in different AAL-related scenarios requiring different degrees of intelligence. With respect to the more challenging scenarios, we also show how biologically-inspired approaches can be used for building more robust systems.

A specific and often occurring problem in an AAL context is people falling in their bathrooms. Here, we use a multisensory approach by combining piezoelectric sensors embedded in the ground with ultrasonic sensors attached to the ceiling. While a piezoelectric sensor can already reliably detect the vibrations caused by a falling person, it is not always possible to distinguish the corresponding pattern from other vibration sources, thus leading to potential false alarms. However, when fusing this information with height measurements from ultrasonic sensors, it is possible to track a person's coarse location and to determine whether the person is standing up or lying on the ground. An addition, detecting more unpredictable emergencies can be achieved by monitoring a person's physical and emotional state. In order to solve this problem, we have developed a set of wearable bio-sensors that continuously measure the person's heart rate, skin-conductance level, and temperature. Based on this data it is possible to estimate a person's stress level and determine what level of assistance is currently required.

Activity monitoring does not need to be confined to recognizing human activities. The aim of another project we are engaged in is to monitor the behavior of ships off the Dutch coast in real time in order to detect emergencies and malicious acts. The main issue here is to detect anomalies by integrating information from many heterogeneous sources, each associated with different levels of uncertainty and trustworthiness. Since the information is provided at different levels of granularity (ranging from low level sensor readings to high-level descriptions of domain ontologies), a very general framework is required for fusing the information adequately. Here, we use Dempster-Shafer theory as a means to bridge the conceptual gap between statistical sensor models and high-level constraints provided by experts. This approach allows the system to distinguish between different types of uncertainty and their origins (e.g., randomness, imprecision, lack of information), which in turn allows operators to make more informed decisions.

Finally, we present ongoing research on monitoring the daily activities of people in an AAL scenario based on biologically-inspired principles for providing assistance and detecting abnormal behavior. These principles and the corresponding architecture have already been successfully applied to the problem of self localization and object recognition, both problems that play an essential role in activity recognition. A key limitation of most technical systems in non-restricted AAL environments is their use of very limited models of the world that capture only a small fraction of the relevant information. This is in stark contrast to the complexity and diversity of representations used by biological cognitive systems to understand their world and are able to reliably recognize the actions and intentions of others (as can be observed in mirror neurons). In order to accomplish this highly difficult task, they have to integrate a wealth of diverse information originating from many different sources. This includes bottom-up processing of multisensory stimuli as well as the incorporation of top-down context knowledge related to cognition. In our system, the bottom-up information is provided by multiple wearable sensors, including a camera for scene analysis. Processing all this information in real time requires powerful tools for information reduction. This is achieved by modeling principles found in biological vision, in particular with regards to the detection of relevant features (nonlinear filters derived from natural scene statistics), luminance adaptivity (ratio of Gaussians), and feature coding (linear filters and neural gain control mechanisms). These extracted features are directly combined with corresponding motor information at different levels of granularity, leading to a sensorimotor description of the environment.

Another problem that biological systems successfully solve is the intelligent use of all the available information. In our system, the selection of features is not only driven by the bottom processing but also influenced by top-down processes, which provide context knowledge about the current situation derived from statistical models and domain ontologies. This top-down control is modeled as an information theoretic problem in our system. Based on the current belief about the world (e.g., ongoing activities), the systems aims to acquire the most relevant information by maximizing the expected information gain, leading to more efficient processing of all the potentially available information (an example of this in humans are saccadic eye movements).

Overall, we believe that technical systems for activity recognition can benefit from applying principles found in biological cognitive systems. While some problems in an AAL context can certainly be solved by established approaches, once these problems become less restricted, biological systems are still vastly superior regarding robustness and adaptivity. The work described here is only a first step in this direction, however, it does already incorporate some of the features that would be required for building more general architectures closer to the biological standard.

-> Download presentation