A Focused, Context-Sensitive Approach to Monitoring

1. The Monitoring Problem

Timely detection of anomalous behavior is essential for the continuous safe operation and longevity of aerospace systems. The pilot of ajet aircraft must be aware of any conditions which may affect thrust during the critical moments of takeoff. The thermal environment onboard Space Station Freedom must be carefully controlled to provide uninterrupted life support for the crew. The Mars Rover must react quickly to an unpredictable envi­ ronment or the mission may come to an abrupt conclu­ sion.

Monitoring a physical system involves a number of problem-solving tasks. Dvorak, in his survey of work on expert systems for monitoring and control [Dvorak 87], lists among these tasks recognizing abnormal conditions, combiniQg sensory information into a picture of the glo­ bal state of a system, isolating faults, predicting both normal and faulted behavior, and maintaining safe oper­ ation in the presence of faults. In addition, decisions for each sensor are determined ahead of time by analyz­ ing the designed nominal behavior for the system. Whenever a sensor value crosses a threshold during op­ eration, an alarm is raised.

The problem with this approach is that the nominal be­ havior of even moderately complex systems often de­ pends on context. For example, an earth-orbiting space­ craft periodically enters and emerges from the Earth's shadow. Impingent solar radiation changes the thermal profile of the spacecraft, as does the configuration of currently active and consequently, heat-generating sub­ systems on board. Thresholds on temperature sensors should be adjusted accordingly. A particular tempera­ ture value may be indicative of a problem when the spacecraft is in shadow or mostly inactive, but may be within acceptable limits when the spacecraft is in sun­ light or many on-board systems are operating.

Fixed alarm thresholds are useful for defining the operat­ ing limits of a physical system, such as the point of over­ balance of a rover, or the temperature at which, say, the onboard computer of a spacecraft is at risk. Nonetheless, they are woefully inadequate for verifying the nominal operation of a system with many operating modes, or one which interacts with an environment The problem is that fixed alarm thresholds are derived from an over-summarized model of the behavior of a system. If the thresholds are chosen conservatively, then false alarms occur. If they are chosen boldly, then undetected anomalies occur. What is needed is a capability for ad­ justing alarm thresholds dynamically. Alarm thresholds should be chosen according to expectations about the nominal behavior of a system as it changes in different operating contexts. Later on in this paper, we present our approach to dynamic alarm threshold adjustment based on causal simulation of the device.

Another issue which arises in monitoring concerns how to best utilize available sensors to efficiently and reli­ ably, but not necessarily comprehensively, verify the nominal operation of a physical system. Just as the nominal values in a system being monitored depend on context, so do the subset of sensors, which can most di­ rectly verify those values depend on context. The famil­ iar activity of driving an automobile helps to illustrate this idea. A variety of sensors are provided to the opera­ tor of an automobile: fuel gauge, temperature gauge, speedometer, several mirrors, etc. However, the driver does not use all of these diverse sensors all of the time. The speedometer may be checked periodically, or when a speed limit sign is passed; the right-side mirror is prob­ ably only used during lane changes. There are two points to be made: one concerns relevance, the other

concerns resources.

Individual sensors are appropriate for verifying only some small, localized subset of the possible behavior of a system. The choice of which sensors to sample and in­ terpret at any particular time should be based on expec­ tations of what is to happen in the system and, perhaps, how it is to interact with an environment. However, even after a suitable subset of the available sensors is identified, there may not be the resources available, whether human or machine, to sample all the selected sensors and interpret the data within a required response frame. What is needed is a capability for assessing the importance of predicted events, so that while it may not be possible to comprehensively verify the expected be­ havior of a system, still the most reliable verification within available resources can be performed.

An illustration of the need to focus attention in monitor­ ing comes from the jet aircraft domain. Some of the re­ cent commercial aircraft catastrophes have been attribut­ ed to insufficient thrust during the critical moments of takeoff. There are many possible indicators of low thrust available to a flight crew. For example, a low ex­ haust gas temperature in an engine may produce reduced thrust Also, a low turbine fan rotation speed in an en­ gine may imply reduced thrust, because fuel input is based partly on this parameter. The challenge is to di­ rect the attention of the flight crew towards information useful for planning actions in real time without over­ whelming them.

A monitoring strategy must take into account the reality that not all sensors should or can be checked all of the time. As the operating context of the physical system being monitored changes, the collection of sensors which provide the most immediate information on the state of the system also changes. Further on in this paper, we present our approach to sensor planning in monitoring. We describe a method for assessing the im­ portance of predicted events in a system, based on rea­ soning about causal dependencies among events, and about how events relate to intended goals of the design­ ers or operators of a system.

3. Other Work

Within NASA, there are other projects underway in which the goal is to develop a monitoring and a diagno­ sis capability for aerospace systems. Among these is the KAlE project at the Kennedy Space Center, whose do­ main is the Shuttle Liquid Oxygen Loading system [Searl et al 88]. In this project, causal models are used to support sensor validation, fault diagnosis, and the planning of control actions.

The goal of the FAULTFINDER project at Langley Research Center [Abbott 88] is to develop an inflight monitoring and diagnosis capability for jet aircraft These investigators have explored the use of multiple representations and multiple levels of abstraction to be able to reason about diverse faults, to focus attention during reasoning, and to provide accessible information to a flight crew.

Outside of NASA, there have been a number of efforts aimed at developing knowledge-based expert systems for monitoring and control. ESCORT [Sachs et al 86] is a shell for developing real-time expert systems to filter and focus information during plant emergency situa­ tions. REACI'OR [Nelson 82] is an expert system for monitoring nuclear power plants which detects anoma­ lous behavior, assesses the seriousness of the situation, and recommends appropriate actions. REALM [Touchton and Casella 86] is an advisory system which detects and classifies emergencies in nuclear power plants and is able to predict further consequences of those emergen­ cies.

Numerous other examples exist of efforts to develop monitoring and control systems. The reader is referred to Dvorak's excellent survey of the area [Dvorak 87] and to the survey of real-time knowledge-based systems in [Laffey et a/88].

The causal reasoning paradigm, which is at the core of our approach to the monitoring problem, is now a well­ established area of investigation within Artificial Intelligence. The advantages of the causal approach, which involves modeling a system at the level of compo­ nents and mechanisms, include the ability to reason about unforeseen interactions, the ability to reason about dependencies among events, and the ability to generate accessible explanations. The seminal efforts in this area include Forbus' process-centered approach [Forbus 85], de Kleer and Brown's device-centered approach [de Kleer and Brown 85], and Kuipers' qualitative mathe­ matics approach [Kuipers 86].

In the specific area of monitoring, Dvorak's MIMIC project stands out as the most comprehensive current re­ search effort [Dvorak and Kuipers 89]. Dvorak creates a component-connection model of a system and employs the QSIM qualitative simulator [Kuipers 86] to generate expectations about the system's nominal behavior. An inductive learning method is used to create a set of symptom-fault rules for known faults, and these rules support the formation of fault hypotheses whenever sensor data does not match predictions from the causal model. When anomalous behavior exists, several fault models can be tracked in parallel until one emerges as the hypothesis with the most explanatory power. The ability to continue tracking a faulted system is important because large, complex systems almost always contain faults and the challenge is to maintain safe operation in the presence of faults.

4. The Approach

At the center of our approach to addressing the two is­ sues of dynamic alarm thresholds and sensor selection is a causal model of the system being monitored and possi­ bly, its environment Simulation of this model directly solves the problem of alarm threshold adjustment. Predicted values and their time tags indicate how and when to alter the alarm thresholds associated with sen­ sors so that they reflect expectations about the nominal operation of the system in changing contexts.

Another result of simulation is information about causal dependencies among predicted events of a system. This information is used to assess the importance of individu­ al events. Briefly, the most important events are taken to be those which either cause or are caused by the greatest number of other events. An ordering on predicted events reflecting this causal notion of importance serves as the basis for allocating sensor resources to selectively verify the expected behavior of a system [Doyle et a/ 87].

In the remainder of this section, we describe (1) the ar­ chitecture of our predictive monitoring system, called PREMON, (2) what our causal models of physical systems look like, and how they are simulated, and finally, (3) our approach to sensor planning, based on analyzing causal dependencies.

4.1 Architecture

There are three modules in the PREMON system: a causal simulator, a sensor planner, and a sensor interpreter. See Figure 1.

The causal simulator takes as input a causal model of the system to be monitored, and a set of events describing the initial state of the system and perhaps some future scheduled events. The causal simulator produces as out­ put a set of predicted events, and a graph of causal de­ pendencies among those events.

The sensor planner takes as input the causal dependency graph generated by the causal simulator and determines which subset of the predicted events should be verified.

These events are passed on to the sensor interpreter.

The sensor interpreter compares expected values as pre­ dicted by the causal simulator with actual values from sensors. Alarms are raised here when there are discrep­ ancies. Finally, the most recent sensed data is passed back to the causal simulator to contribute to another pre­ dict-plan-sense cycle of monitoring.

4.2 Causal Models and Causal Simulation

We represent physical systems as a collection of quanti­ ties and mechanisms. Quantities are continuous parame­ ters such as temperature, position, and amount-of-stuff. Quantities are specified by a physical object, a type, and an order. Examples of quantities are {HEATER TEMPER­ ATURE RATE} and (SWITCH POSffiON AMOUNT}.

Events describe discontinuous changes in the value of a quantity. Events are specified by a quantity, a value, and a moment. Examples of events are {HEATER TEMPERA­ TURE RATE POSITIVE 61} and {VALVE-17 POSffiON AMOUNT OPEN 0}.

Mechanisms capture causal relations between quantities. More specifically, they describe how a change in one quantity results in a change in another quantity. Examples of mechanisms are HEAT fLOW, THERMAL EX­ PANSION, LATCH, and GRAVITY. A mechanism is speci­ fied by a time constant, a distance, a sign, an efficiency, a bias, an alignment, and a medium. Figure 2 shows the representation of a HEAT fLOW mechanism.

A causal model then, consists of a set of quantities and a set of mechanisms between those quantities. A causal model can be represented by a graph where the nodes are quantities and the arcs are mechanisms. Simulation of a causal model involves predicting new events, via mecha­ nisms, from known or previously predicted events. The simulation method outlined in the next few paragraphs is described more fully in [Doyle 88, 89].

When the quantity named in an event appears as the cause quantity in a mechanism, a new event is predicted as follows: (1) the quantity of the new event is the effect quantity of the mechanism, (2) the value of the new event is computed from the value of the given event and the sign and efficiency of the mechanism, (3) the mo­ ment of the new event is computed from the moment of the given event and the time constant and distance of the mechanism, and (4) the new event occurs only when constraints specified in the bias, alignment, and medium of the mechanism are satisfied. The bias of a mecha­ nism specifies constraints on directions of change. For example, current through a wire can cause it to heat up, but not to cool down. The alignment of a mechanism specifies constraints expressed as inequalities. For ex­ ample, heat flow is from the warmer to the cooler site. The medium of a mechanism is a physical connection such as a wire, a pipe, a linkage, etc. The predicted ef­ fect occurs only when the specified physical connection is in place.

In Figure 2, a typical event is shown, this one describing a temperature change. The HEAT FLOW mechanism is used to predict another temperature change event.

Simulation would be straightforward if physical systems could be modeled exclusively as simple mechanism chains between input and output quantities. However, some mechanisms serve to enable or disable other mech­ anisms, such as a valve controlling a fluid flow, or a latch inhibiting the transmission of motion through a me­ chanical coupling. In these cases, the contributions of the separate mechanisms combine multiplicatively. The contributions of separate mechanisms also can combine additively, as when two fluid lines empty into the same container, or two opposed forces produce an equilibrium state. The details of our simulation method for interact­ ing mechanisms are described in [Doyle 88,89].

4.3 Sensor Planning

The output of the causal simulator is a trace of predicted events and the dependencies among them. The depen­ dencies are derived from the mechanism structure of the system. A dependency between two events is a record that there is a mechanism in the system which causally relates the events.

Analysis of the causal dependencies in a simulation trace supports decisions about which events to monitor. In our approach, the importance of events is assessed by determining how many other events are effects or causes of a given event. In other words, the importance of an event is related to the amount of subsequent activity it supports and the amount of activity which supports its occurrence. Critical events which lie on several causal paths between inputs and outputs should be verified with care, perhaps with a battery of sensors. On the other hand, events which are side effects and do not support further activity in the system may be ignored complete­ ly. See Figure 3.

This analysis method weights all dependencies in a caus­ al graph equally. Several criteria might form the basis of a non-uniform weighting scheme. For example, a priori or empirical knowledge about probabilities of failure might bias the allocation of sensor resovrces towards those components in a system known to be unreliable. Similarly, parts of a system where redundancy has been built in might be given less careful attention than other parts.

Our causal analysis method for determining what subset of predicted events to monitor is similar to the minimum entropy method of [de Kleer and Williams 87] for deter­ mining the site of the most useful next measurement in troubleshooting. Their technique involves propagating observed values and failure probabilities along a causal dependency graph for a circuit.

5. An Example: The JPL Space Simulator

The JPL Space Simulator is an environmental chamber in which spacecraft and instruments can be subjected to some of the aspects of the space environment: intense cold, near vacuum, and solar radiation.

A mirror is used to direct simulated solar radiation onto the spacecraft or instrument inside the chamber. This mirror must be cooled separately from the shroud which surrounds the chamber to compensate for the additional radiation falling on it. Cold gaseous nitrogen is used as the cooling medium and is circulated by a fan. Chilling is achieved by injecting liquid nitrogen into the gaseous nitrogen. Warming is achieved through an electrical heater. A causal simulation of this cooling circuit is shown in Figure 4.

Using the causal analysis technique outlined above, the flow of gaseous nitrogen at the fan is identified as the single most critical event in the predicted nominal be­ havior of the circuit. This event affects gas flow around the entire circuit and indirectly, heat flow around the en­ tire circuit. The only events unaffected by this event are the source temperature changes at the chiller and heater. This result of causal analysis captures the intuitive no­ tion that nothing at all happens in the cooling circuit if the fan stops operating. Other important events in the predicted operation of the circuit are the temperature changes at the chiller and heater. Measurements made at these sites also provide informative feedback about the nominal operation of the circuit.

This example has _been implemented and illustrates our current causal simulation and sensor planning capabili­ ties. We are beginning to target our developing predic­ tive monitoring capability to other aerospace systems ex­ isting or being designed within NASA. Potential do­ mains include power and thermal distribution systems for planetary rovers, ground antenna control systems in the Deep Space Network, and subsystems onboard earth­ orbiting spacecraft

6. Future Work

Our investigations so far have clarified and uncovered other research issues which also need to be addressed. In this section, we briefly outline these issues.

6.1 Generation of Fault Hypotheses

Because complex systems rarely operate completely fault-free, a monitoring system should include also the capability to generate fault hypotheses and to incorpo­ rate fault models into the current model of a system so that behavior can continue to be predicted and tracked in the presence of faults. During the diagnostic process, it may be necessary to track several models in parallel, each corresponding to a different fault hypothesis.

6.2 Selective Simulation

We have already made a case for selective use of sen­ sors. In this vein, just as there may be insufficient resources to interpret profusive sensor data, there may be also insufficient resources to conduct a comprehensive simulation of a system. Decisions about which part of a system to simulate also are likely to be context-depen­ dent. Selective simulation interacts with sensor plan­ ning in the following way: When simulation cycles are shifted from one part of a system to another, an overhead of sensor reads is needed to determine current state, in order to bootstrap the new focus of simulation.

6.3 How Far Ahead to Predict

Another way of dealing with the problem of limited re­ sources for simulation is to constrain the number of events predicted in a pass through the causal simulator. The tradeoff is between generating enough of a causal dependency graph to drive sensor planning and main­ taining a real-time predict-plan-sense monitoring cycle. Another potential factor is ambiguity in simulation. Particularly when a system model is qualitative, there may be insufficient a priori information to determine which of several alternate states a system may enter. A direct way to counteract such branching is to suspend simulation and utilize explicit sensor reads to pin down ambiguous states.

7. Conclusions

Detecting anomalies in the operation of a system is a dif­ ficult problem when the behavior of the system is complex or involves interaction with an environment, and when the number of sensor channels is large. Under these conditions, nominal values and the most informa­ tive sensor data change according to context We have addressed two specific issues in monitoring: how to ad­ just alarm thresholds dynamically, and how to verify be­ havior selectively but reliably. At the center of our ap­ proach to solving both problems is the use of a causal model of the system being monitored. Simulation of a causal model serves both to generate expectations about nominal sensor values, and to provide dependency infor­ mation useful in assessing the importance of predicted events and in allocating sensor resources accordingly.

The key idea in this paper is letting go of the notion of comprehensive monitoring. More likely than not, there will be insufficient resources for predicting behavior and interpreting sensor data. In the face of this limitation, our emphasis is on verifying the operation of a system efficiently and reliably, by carefully focusing computa­ tional resources to gather the most informative, if incom­ plete, feedback on nominal operation within changing contexts.

Acknowledgements

The content of this paper has been enhanced through dis­ cussions with Kathy Abbott, Dan Dvorak, Lorraine Fesq, David Franke, Ben Kuipers, and Ethan Searl. The ex­ pertise of Bob Lane was crucial in developing the model of the mirror cooling circuit

The research described in this paper was carried out by the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration.