Abstract This paper presents a hybrid framework for real-time continuous monitoring using Body Sensor Networks. In this framework, we illustrate how domain-specific expert knowledge and observation of data in its feature space are combinedly used for rule construction. A rule-based mechanism is deployed as a tool for organizing/managing recognition models derived using a numerical method. As a demonstration, we construct a knowledge base for automatic activity recognition based on assumptions on characteristics of activities under investigation and constraints on device packaging design and on the way a device can be worn by a user. The framework is implemented using Java Expert System Shell and is applied on datasets collected under a semi-natural condition during a device-orientation-independent and device-location-independent activity recognition experiment. The implemented system is evaluated in situations where device orientations and locations are not known in advance and arbitrarily changed by users. © 2013 ICIC International.