Building Intelligent Middleware for Large Scale CPS Systems Niels Reijers, Yu-Chung Wang, Chi-Sheng Shih, Jane Y. Hsu Intel-NTU Connected Context Computing Center National Taiwan University Taipei, Taiwan Abstract—This paper presents a study on building intelligent middleware on wireless sensor network (WSN) for large-scale cyber-physical systems (LCPS). A large portion of WSN projects has focused on low-level algorithms such as routing, MAC layers, and data aggregation. At a higher level, various WSN applications with specific target environments have been developed and deployed. Most applications are built directly on top of a small OS, which is notoriously hard to work with, and results in applications that are fragile and cannot be ported to other platforms. The gap between the low level algorithms and the application development process has received significantly less attention. Our interest is to fill that gap by developing intelligent middleware for building LCPS applications. The goal for our research is to produce a "zero-cost" deployment capability where an LCPS application does not need to be constrained on what and where sensors are deployed. An intelligent WSN middleware can automatically perform optimal sensor node configuration, bandwidth allocation, fault management, and re-configuration in response to new missions and new devices available. Keywords: wireless sensor networks; machine-to-machine; middleware; virtual machine; configuration; policy; profile

I.

INTRODUCTION

Many existing wireless sensor network (WSN) research projects have focused on low-level algorithms such as routing, MAC layers, and data aggregation. At a higher level, a significant number of single-purpose WSN applications have been developed and deployed. In between, however, the gap between the low level HW/network support and the mission of high-level applications, has received significantly less study. Most WSN implementation projects show that building a complete, working WSN is still very hard, not to mention making them dynamically adaptable or capable of evolving with new missions, new sensors, and/or new target environments. Except for the low level building blocks, there is very little software reuse from one WSN to another WSN. WSNs have historically been built with a strong coupling among applications and the underlying network infrastructure. Many WSN applications are built directly on top of the hardware or on some minimum operating system, such as TinyOS, which is very hard to work with. This results in applications being programmed from the individual node’s perspective, telling a node what to do, rather than allowing the

Kwei-Jay Lin Department of Electrical Eng. and Comp. Sci. University of California, Irvine Irvine, CA, USA

application to express globally what it really wants from the sensor network. Furthermore, since most WSN applications are specifically designed to particular scenarios and unique target environments, most behavior is hardcoded, resulting in rigid networks that are typically not very flexible at adjusting their behavior to faults or changing external conditions. A more high level vision on how large-scale cyber-physical systems (LCPS) or networks of sensors should be programmed, deployed, and configured is still missing. Indeed, we are far from conducting Rapid Application Development (RAD) for LCPS. This project aims to fill the gap, by developing a flexible and self-organizing middleware support for LCPS deployment and management in the envisioned ubiquitous LCPS of the near future. The middleware will pursue intelligent capabilities to detect, identify, configure, deploy, and if necessary, repair, reload and redeploy an LCPS application. The intelligent capabilities have similar goals to autonomic computing, including SelfConfiguration, Self-Healing, Self-Optimization, and SelfProtection. But our research project will be specifically for LCPS where much of computing, network, energy and memory resources may be under severe constraints. We design the middleware architecture as configurable modules so that different subsets of the “self-X” capabilities may be selected and embedded in a WSN middleware instance for a specific sensor node, in order to best meet the needs of the node itself and the LCPS application. The grand vision for our research is that future LCPS should have "zero-cost" deployment where users of an LCPS application do not need to be concerned on how and where to deploy sensors. The built-in intelligence from the proposed LCPS support can automatically perform the optimal sensor node configuration, bandwidth allocation, fault management, and re-configuration in response to new missions and new device deployment. Much like the past transition from lowlevel assembly codes to high-level programming using general purpose OS and compiler support, LCPS programming may be as platform-independent as possible while keep only the most essential system primitives to allow automatic performance optimization.

II.

PROJECT OVERVIEW

We envision that LCPS will have many heterogeneous sensor nodes (shown as various shapes and sizes in Figure 1), and one or more gateway nodes connected to them. The gateway node (shown as EeeBox in the figure) usually has a more powerful computing and networking capability than sensor nodes, and one of the gateways will take the role for configuration decision making, referred as Master in our project, making coordination decisions and configuring the system and sensor nodes. Master may be connected to some Cloud servers in order to maintain some knowledge base for the application and access big computing power if needed. In our current study, we assume there is only one Master in any LCPS. There may be shadow Masters in the system to provide fault tolerance through some real-time backup process. In the future, we will also study distributed Masters for more flexible !"#$%&'()*+',-./0.&' and scalable support on LCPS. '$(%")%&

:%(7;<$"'()(" #$%&"'()("

<$/#&

!"#$%& :)-0$&

#)*+$"'()("

5/0/6/%$&

#$%&"'()("

*++,-./0-"(1& 2$$3"4&

7"")#-(/0")&

#$%&"6*%)+*7"8$99(3$"

#$%&"6*%)+*7"8$99(3$" 891&:$6&9(0$);/.$&

#,*-"),$"&()("*%"./" 0" 1$(+%2%3"

4$5*)$"6*%)+*7" '$=2>$9" !"

#5(+)"'$=2>$9"

Figure 1. A simple example LCPS application

An LCPS application is started by a user who wants to access sensor nodes via Master or other gateways to be configured by Master. When the user initiates the application, Master starts a discovery phase to identify all "capable" sensor nodes within its connection network. A sensor node is "capable" if it is pre-loaded with our light-weight discovery protocol support in order to communicate with Master for responding to its query and configuration requests. Each sensor should be loaded with a device profile definition, identifying its capabilities, operational conditions, data attributes and sensing/actuating actions. After all device profiles are collected by Master, Master will match the logical devices used in an application to the available physical nodes in the current operational environment. Many LCPS applications are heavily influenced by the user preference and system context, since users and applications are often mobile and have changing needs under different situations. For example, a user may need different ambient lightings at different time and locations. Another example is that a smart home should provide different room temperatures during night or day, and also depending on the activity inside of each room. We therefore need a policy framework to make LCPS intelligent and user-centered. In our middleware design, users are able to define some application policy by using friendly tool or language. Master will be informed of the user (or environment) policy for the specific application context. The policy will be used by Master in the configuration decision

process, connecting and setting up sensors, linking them up with gateway nodes and cloud servers in a complete WSN infrastructure. Finally, in most LCPS applications Master will need to configure, optimize, and reconfigure sensors in real time. The configuration decision will be made by an optimization engine residing on Master and may be supported by cloud servers and gateways. We propose to implement the configuration engine using incremental algorithms so that initial configuration can be set up quickly, refined as the application is being used, and eventually stabilizes. Configuration decisions will be made based on factors such as sensor profiles, user policy, context knowledge, application history, and real time status, allowing the application to respond to changing external conditions. III. TRI-FRAMEWORK APPROACH To achieve the above goals we will develop three orthogonal middleware frameworks: 1. Sensor profile framework: to enable the handling of heterogeneous or new sensor nodes, and for high-level, logical abstraction of low-level, native sensor operations. 2. Application policy framework: to allow user-friendly specification of application execution objective, and context-dependent optimization of WSN performance. 3. Real-time decision framework: to facilitate real-time WSN management where decisions are made flexibly and quickly using current data and available sensors. The most important task of Master is to configure, optimize, and reconfigure sensors. To do this it communicates with sensors through a layer of abstraction, hiding hardware and network details of the underlying sensor platform. During the discovery and identification phase, Master uses the profile framework to discover the capabilities of connected sensors, and configure sensors' parameters. While there has been some work on discovery protocols in other contexts (for instance UPnP and DPWS), these approaches are often too heavy for the resource-constrained nodes in a sensor network.

Figure 2. Logical vs. physical devices

We have studied the structure of device profiles for sensors the we want to support initially, and are currently designing a light-weight profile protocol to be used in resource constrained nodes. On the other hand, the profile framework must be general enough to support the type of automatic discovery and configuration we want to achieve. Figure 2 shows the profile matching between logical and physical devices. Each application is compiled into app codes that are connecting or running on logical devices. On the other side of the middleware, physical devices with specific capability profiles are discovered and identified. The middleware must then match the capability of each logical device to that of some physical device. If there is a one-to-one match, the decision is easy. If there are more physical devices than logical devices of compatible capabilities, some of the physical devices will not be used. But if there are fewer physical devices than required logical devices, we will need to consider resource sharing. For example, two logical temperature sensors are needed to run the application in Figure 2, yet only one physical temperature sensor is available. The system needs to use the single temperature sensor for feeding data into two logical sensors. Depending on the device usage from the application, we may need to use data sharing, data coercion, or even sensor time multiplexing to share physical sensors. On the application’s side, a policy framework will be developed to express the application’s requirements such as desirable energy lifetime, data accuracy, system responsiveness, reliability, etc. Policy will be important to drive the configuration optimization, since most WSN operations have some tradeoffs between conflicting goals. For example, a better responsiveness will require more frequent sensor readings, which will consume more energy. Data precision requirement will affect energy consumption and network bandwidth. A good policy engine is needed to produce the optimal WSN configuration under a specific context and mission. In addition, the policy framework can be used to simplify application coding, since runtime decisions such as routing and energy consumption can be produced automatically from the policy engine in the middleware. The configuration decision is to be made by Master which utilizes the knowledge gathered from the other two frameworks and the application context to intelligently configure the WSN. This configuration optimization may require a significant amount of computing power not available to Master. We plan to design a distributed decision framework so that cloud servers, together with Master, can run a distributed optimization algorithm in order to reach a faster or better configuration decision. Master, with the help from cloud servers, will also continuously monitor the network and its own health and reconfigure WSN in response to faults or changing conditions. Thus, our middleware is regarded as a Virtual Middleware (VMW). The reasons for this are twofold. First, as sensor networks become more wide-spread, it is increasingly likely that applications will want to take advantage of different types of sensors available in a given area. Having a virtual sensor will allow applications to run on heterogeneous networks of sensors. Second, when Master decides to reconfigure the network, the process of reprogramming nodes will be easier using a virtual machine design so that code updates can be

smaller since the virtual sensor can offer higher level primitives specific for LCPS applications. IV. RELATED WORK There are projects that have addressed some parts of our goal in this project. For example, related to the policy aspect of WSN is the ADAE system [1] developed at the IT University of Copenhagen. ADAE configures the network according to a policy describing the desired data quality, including fallback options which the system may use in case the ideal situation cannot be achieved. ADAE also reconfigures the network in response to changing conditions such as node failures, unexpected power drops, or interesting events detected by the network. However, the language used to describe the policy and constraints is hard to use and it’s likely a skilled engineer is still needed to translate the user’s requirements into the constraint optimisation problem ADAE uses as input. One major effort of our project is to design and implement a simpler policy framework with dynamic VMW support. There have been several projects which have developed virtual machines for very resource constrained devices in general, or for sensor networks specifically. Designed specifically for WSN, UC Berkeley’s Maté [2] was one of the first to prove VMs can be made to run on sensor nodes. Recently ETH Zürich has developed a more powerful VM called SwissQM [3]. It is based on a small subset of the JVM instruction set, extended with several sensor network specific instructions. In both systems however, the application has to be programmed in very low level, assembly-like language. Several VMs for more high level languages like Python or Java have also been developed, which fit on severely resource constrained nodes. Almost all rewrite the original bytecode to their own format and employ various techniques to reduce code size. Some functionality is sacrificed in order to fit on the sensor nodes, for instance reflection is typically not supported. The Python-on-a-chip project [4] developed a Python bytecode interpreter small enough to fit on sensor nodes. Requiring 55K of program memory and a recommended 8K of RAM, it fits on many, but not all of the current sensor boards. Darjeeling [5] from Delft University of Technology is able to run modified Java bytecode on sensor nodes, with only minor restrictions. It has no support for floating point, 64-bit datatypes, reflection or synchronised method calls, although the later is not really a restriction since synchronised blocks are supported. Similarly TakaTuka [6] is a slightly more complete and more complex JVM, also running on both AVR and MSP based sensor nodes and offering it’s own debugger. Sun’s Squawk VM [7] is significantly larger than Darjeeling and TakaTuka, but offers full CLDC [8] compliance. The opposite approach is taken by NanoVM [9], which takes a minimal approach, trading support for a large number of java opcodes for simplicity and code size. The whole VM fits in the 8K flash of an ATMega8 CPU, leaving 512bytes of EEPROM and 75% of the CPU’s 1K RAM to the application. It’s small size and simple design make it a suitable choice to build our initial prototypes on. While these technologies allow the application to be developed in a higher level language, and to be deployed on

heterogeneous networks, applications still have to be written in an imperative way, and from the node’s perspective. These technologies will serve as a basis on which we build our middleware. We plan to enhance the VM design with the policy framework so that WSN applications can be coded in a very high level manner, similar to a script language for many new application domains. Very different approaches are taken by Agilla [10] and MagnetOS [11]. In Agilla programs consist of software agents that can move around autonomously in the network. While this may allow some behaviours to be expressed in a more natural way, the fact that this programming paradigm is very different from what most conventional languages, and the instruction set which is based on Maté’s assembly like programming, make it hard to use, and unsuitable as a model to build our VMW on. Finally Cornell’s MagnetOS, is interesting because it proposes a novel programming model which supposedly allows the user to write the application as a single Java application, not explicitly related to individual nodes. The system then automatically partitions the applications into pieces (by default along object boundaries), and places these pieces on nodes in the network in such a way as to minimise energy consumption. However, it requires nodes significantly more powerful than what we expect to find in a typical WSN. We consider this to be a possible future direction for our study if and when all future WSNs are proved to be more powerful. On the profile side, sMap [12] provides an attractive and flexible RESTful interface through which we can discover what sensors are available at a certain node and get or set several configuration properties. Although the authors succeeded in dramatically reducing the footprint, it is still relatively resource intensive. It is also limited in the number of properties it exposes, but the idea could easily be expanded to include more diverse node types and configuration options. ZigBee’s approach [13] is less resource intensive, and offers a more complete description of a node and can be applied to more devices. However it is less flexible since it works with a fixed set of device descriptors. In the Z-Wave community, a device class is used to define device profiles. Each profile contains a set of required and recommended command classes. Z-Wave defines a device into a profile which lists the device class and all supported command classes. The profiles are quite generic and fixed. V.

CONCLUSION AND FUTURE WORK

In this paper we have briefly introduced the design of an intelligent middleware for LCPS applications. The project consists of three major frameworks: profile, policy, and realtime decision framework. Since the goal for the project is to provide middleware that can support a wide range of applications, we will consider different application scenarios during its development. These scenarios should differ enough to provide insight on how different environments affect the requirements for middleware. For instance healthcare, environmental monitoring or military

applications all place very different demands on the middleware supporting them. Some of these scenarios will be implemented in our project to verify our design, whereas others will just be used as a design exercise to test our ideas. One of the scenarios we are implementing is called Flow@Home. In this scenario we track the flow of people, pets or objects in a home or office building. This scenario allows us to integrate different types of sensors, such as motion, infrared, sound, light, temperature, RFID, etc, making it an interesting user case for study. At the same time it is a useful building block using which many different kinds of applications, like security systems, babysitting assistance, senior citizen home care, and energy saving, can be built. VI. ACKNOWLEDGEMENT This research was partially supported by National Science Council of Taiwan, National Taiwan University and Intel Corporation under Grants NSC99-2911-I-002-201, 99R70600, and 10R70500. REFERENCES [1] Marcus Chang and Philippe Bonnet, “Meeting Ecologists’ Requirements with Adaptive Data Acquisition” in Proceedings of the 8th ACM Conference on Embedded Networked Sensor Systems (SenSys 2010) [2] Philip Levis and David Culler, “Maté a tiny virtual machine for sensor networks” in Proceedings of the 10th international conference on Architectural support for programming languages and operating systems (ASPLOS X) [3] René Müller, Gustavo Alonso, and Donald Kossmann, “A virtual machine for sensor networks” in Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems (EuroSys 2007) [4] Python-on-a-chip http://code.google.com/p/python-on-a-chip/ [5] Niels Brouwers, Koen Langendoen, and Peter Corke, “Darjeeling, a feature-rich VM for the resource poor” in Proceedings of the 7th ACM Conference on Embedded Networked Sensor Systems (SenSys 2009) [6] Faisal Aslam, Christian Schindelhauer, Gidon Ernst, Damian Spyra, Jan Meyer, Mohannad Zalloom, “Introducing TakaTuka: a Java virtualmachine for motes” in Proceedings of the 6th ACM Conference on Embedded Networked Sensor Systems (SenSys 2008) [7] Squawk Java VM http://labs.oracle.com/projects/squawk/squawkrjvm.html [8] Java Connected Limited Device Configuration http://java.sun.com/products/cldc/ [9] NanoVM http://www.harbaum.org/till/nanovm/index.shtml [10] Chien-Liang Fok, Gruia-Catalin Roman, and Chenyang Lu, “Rapid Development and Flexible Deployment of Adaptive Wireless Sensor Network Applications” in Proceedings of the 25th IEEE International Conference on Distributed Computing Systems (ICSCS 2005) [11] Hongzhou Liu, Tom Roeder, Kevin Walsh, Rimon Barr, and Emin Gün Sirer, “Design and implementation of a single system image operating system for ad hoc networks” in Proceedings of the 3rd international conference on Mobile systems, applications, and services (MobiSys 2005) [12] Stephen Dawson-Haggerty, Xiaofan Jiang, Gilman Tolle, Jorge Ortiz, and David Culler, “sMAP: a simple measurement and actuation profile for physical information” in Proceedings of the 8th ACM Conference on Embedded Networked Sensor Systems (SenSys 2010) [13] ZigBee Alliance, http://www.zigbee.org/