A Qualitative-Fuzzy Framework for Nonlinear Black-Box System Identification

1 Introduction

The problem we address here is how QR techniques can be used to improve the performance of non-parametric approaches to nonlinear black-box System Identification (s1). Recently, due to the paucity of directly applica­ ble results, nonlinear SI has received more and more attention in the control community with a consequent development of a number of new approaches capable to describe the nonlinear dynamics of a real system from input-output data. Neural networks, multi-variate splines and fuzzy logic systems are the most known ap­ proximation schemes used for learning an input-output relation from data [Jang, 1993; Khannah, 1990; Wang, 1994]. Although these approaches are successfully ap­ plied to a variety of domains, they are affected by two main drawbacks: first, the model identification proce­ dure usually requires a large amount of data and is of­ ten extremely inefficient; second, the identification re­ sult, a nonlinear function, does not capture any struc-

Raffaella Guglielmann

Istituto di Analisi Numerica

C.N.R.

via Abbiategrasso 209 27100 Pavia, Italy Liliana Ironi Istituto di Analisi Numerica C.N.R. via Abbiategrasso 209 27100 Pavia, Italy

tural knowledge. With the goal to overcome these draw­ backs, we propose a novel method which combines the qualitative and non-parametric modeling frameworks. Therefore, such a method is applicable whenever the incompleteness of the available structural knowledge of the system under study is not so strong as to pre­ vent from formulating a qualitative model of its dy­ namics through Qualitative Differential Equations ( QDE) [Kuipers, 1994].

We believe that the efficiency and robustness of non­ linear black-box SI methods may improve only if they incorporate and exploit all available knowledge of the system, namely the structural and human knowledge, and the experimental one. Qualitative models and lin­ guistic rules represent properly the structural and human expert knowledge, respectively. As qualitative modeling formalism we have chosen QSIM [Kuipers, 1994] because of both its expressive power to represent QDE and its reasonable predictive capacity. As nonlinear identifiers we have chosen fuzzy logic systems since various classes of FUzzy Systems (Fs) can be proved to have the uni­ versal approximation property [Wang, 1994]. A clear advantage of using FS's deals with their capability to incorporate in the same framework both linguistic de­ scriptions of the unknown system dynamics, in the form of IF-THEN rules, and experimental data. Moreover, the meaning of their parameters is linked to the input-output data. FUzzy inference procedures are initialized with a rule-base which defines the structure of the input-output relation where parameters occur. When such a base is not available or poorly defined, also the fuzzy inference procedures may become extremely inefficient as the in­ ference structure has to be determined using only numer­ ical evidence. The method we propose aims at solving the central problem of the construction of a meaningful rule-base: Fuzzy Rules (FR) are automatically gener­ ated by encoding the knowledge of the system dynamics captured by its qualitative simulated behaviors.

Such a method, which we label FS-QM, may be applied to a number of different domains. As a benchmark we have considered problems from the medical domain. In this paper we discuss the application of FS-QM for the identification of the kinetics of Thiamine (vitamin B 1) and its phosphoesters in the cells of the intestine tissue.

2 Fuzzy System identification

Fuzzy Systems possess good properties as approximators of continuous functions [Wang, 1994) and have been suc­ cessfully applied in the identification and control of non linear systems [Barada and Singh, 1998). Herein, we use FS's to approximate continuous functions from !Rn to !R: the value of an output variable y defined on V C !R is inferred by using a Fuzzy Rule Base (FRB) with n input variables defined on ui c !R, such that u = ul X ... X Un.

As it is well known, the approximation properties of the FS depend on the choice of the operators that define: - the fuzzification of FRB antecedents (or inputs), that transforms a real number Xi E Ui into a fuzzy set with membership function defined over Ui;

- the fuzzy inference engine (FIE) , that maps the fuzzy sets in U into fuzzy sets in V through the FRB;

-the defuzzification of FRB consequent (or output), that transforms into a real number the output of the applica­ tion of the FIE.

In this paper we have exploited the singleton fuzzifier, the product-inference rule [Wang, 1994) and the Cen­ ter Average Defuzzifier [Mamdani, 1974). Through these choices, it is possible to write the resulting FS in func­ tional form as:

where

M represents the number of FR's, while J.L1 is the mem­ bership function associated with the linguistic variable of Xi that appears in the j-th rule, and Yi is the point in V where the correspondent membership function reaches its maximum value.

In the following, we will exploit the class of FS with Gaussian membership functions, so that:

Such a choice allows us to interpret the nonlinear func­ tion approximation problem with FS as the process of identifying the vector of parameters (Y j' x{, a{) of a known nonlinear function from a set of data. More­ over, the FS defined by the equation (1) holds the univer­ sal approximation property [Wang, 1994). The function approximator y(J<.) is dependent on a parameter vector fl., such that y(J<.) = y(;£, fl.), where fl. = {Q, i&} with f; = {yj}, i = {x{} and fl.= {a{}, where i = 1, ... ,n, I= 1, . .. ,M.

Numeric values, which express the prior knowledge in­ cluded in the system, and which provide for an initial guess of the system dynamics, are assigned to the pa­ rameter vector fl. in the construction phase of the FS. Given a set of experimental data, the estimate of fl. is then refined through optimization procedures.

In FS-QM, the description of the system dynamics is performed through rules that give the next value Yk+l of the system output as a function of the values of the current inputs (Y.k) and output (yk) · Then, the output behavior can be described with a Non-linear AutoRe­ gressive eXogenous input model (NARX) of the kind:

where k is a discrete time index, J<.k = {Y.k, yk}, Vk is the measurement error and y(·) has the same meaning of equation (1). Since the model (2) is nonlinear in the model parameters, it is necessary to resort to nonlin­ ear identification techniques to estimate the vector fl.. Within the FS-QM framework a number of approaches may be considered. Herein, the FS is represented as a feedforward neural network, as described in [Wang, 1994), and the identification of fl. is performed by using the Back-Propagation (BP) technique. Such a technique allows us to estimate all the FS parameters through an iterative search in the solution space by employing a gra­ dient descent search. If not properly initialized, such a search procedure may be either trapped at a local mini­ mum or converge very slowly: FS-QM provides for a sig­ nificant initialization of BP algorithm, with a consequent fast convergence to the global minimum.

3 The FS-QM method

Fuzzy set theory and QR techniques share the motiva­ tion of facing too complex or ill-defined physical systems to be analyzed through conventional techniques. Then, the idea of unifying both frameworks with the goal of producing effective solutions to an extended range of application problems is a matter of course [Shen and Leitch, 1993; Vescovi and Trave-Massuyes, 1992]. Such works aim at developing qualitative simulation tech­ niques which are enriched by a fuzzy description of quan­ tities and functional relations.

Our method, FS-QM, is grounded on the intuition that analogies between elements of the QSIM and fuzzy for­ malisms can be highlighted (Fig. 1).

At a first level, namely the modeling one, given a phys­ ical system S characterized by {Xi, i = 1, ... , m} state variables, the quantity space Li associated with the vari­ able Xi may find its semantic correspondence with the set of the linguistic values defined over the universe of dis­ courseUiofxi. ThesetLiu{(lj,lj+l}, lj E Li}j=l ,j ' -l , where j' is the cardinality of Li, contains all signifi­ cant distinct qualitative magnitude values (qmag) of Xi, whereas the linguistic values associated with Xi are rep­ resented by fuzzy sets characterized by the membership functions {J.Lf,i' = 1, . .. ,2j'-1}.

At a second level, more related to the simulated sys­ tem behavior, the qualitative system dynamics, which is described by the simulated Qualitative Behaviors ( QB), may find its correspondence in fuzzy rule-bases. Each QB is defined by a sequence QS(S, tk) of qualitative states of the system, where QS(S, tk) is am-tuple of the qual­ itative values, QV(xi, tk), of each Xi· Given the input and output variables, all their possible dynamics are ex­ tracted from the behavior tree, suitably manipulated, and automatically mapped into rule sets where the input and output variables are, respectively, the antecedents and consequent in the IF-THEN rules.

Remark 1. Let us observe that we map a quantity space, which is a set made up of landmarks and intervals, into fuzzy sets: then, apparently, we map a non-uniform set into a uniform one. Landmarks are symbolic names denoting particular real numbers that separate qualita­ tively distinct regions. However, in many applications, and in particular the medical domain we are interested in, a landmark value may not have a crisp representa­ tion but, in its turn, be defined by an interval with "soft" boundaries: as a matter of fact, we consider this case, and then we map a set made up of intervals into fuzzy sets.

The overall system identification procedure proceeds in three main phases:

QSIM model formulation and simulation, and definition of the fuzzy elements. The prior structural knowledge of the system at study must be organized so that its be­ havioral model can be defined. More precisely, the vari­ ables of interest and the network of interactions between them, along with their mathematical descriptions, must be specified: variables are described by their respective quantity spaces, whereas their interactions by a set of qualitative constraints which include both functional de­ pendencies and equations governing the system dynam­ ics. Then, an initial state of the system, which may describe a perturbation on it , has to be provided to sim­ ulate its behavior. An attainable envisionment, which does not generate any new landmarks, is performed to produce in one run all possible behaviors that could fol­ low from the given possibly incomplete specification, and shows us the entire range of possible system dynamics at once. The idea underlying the mapping of a QB into a set of FR's exploits the semantic correspondence between the elements of quantity spaces and linguistic values defined through fuzzy membership functions p,. Then, the fuzzy quantity spaces of the input-output variables, i.e the lin­ guistic variables associated with the quantity spaces of the input-output variables, and their corresponding p,, have to be defined. Such definitions are suggested by the expert knowledge.

Construction of the FS. The prior knowledge of the dynamics of the system captured by the QB's derived at phase 1 together with the fuzzy quantity spaces are exploited to generate automatically a base of FR's whose antecedents and consequent are the input and output variables used to quantitatively identify the dynamics of the system. The choice of the FIE, the fuzzification and defuzzification methods, complete the definition of the FS.

Identification of the FS from the experimental data. The generated rules, interpreted in accordance with the FIE selected at the previous phase, initialize an optimiza­ tion procedure for the identification of the parameters in the FS, which learns from the available experimental data an accurate input-output relation.

3.1 Construction of the FS

Given n input variables Xi (n ::=; m), and the output variable y, a QB is automatically mapped into a FRB in the following steps of the method:

1. from the time set T, draw out T = Ui=I Tx, U Ty , whose elements are the significant time-instants of both Xi and y (Tx, and Ty are the sets of distinguished time­ points of Xi and y, respectively);
2. from T, build T ={to} U {(tj-1, tj),tj}j=l.. .k, where k is the cardinality ofT:
3. Vtk E T, where tk may be either a time-point or a time-interval, repeat:
4. (a) consider QS(S, tk) ;
5. (b) from QS(S, tk), draw out the qualitative values of input and output variables, namely QV(xi, tk) and QV(y,tk) (clearly, QV(y,tk+I) if {xi} n {y} =f. 0);
6. (c) from QV(xi, tk) and QV(y , tk), draw out its related qualitative magnitude qmagi', i.e. qmagi' (xi, tk) and qmagi'(y, tk);
7. (d) consider the linguistic values, and therefore the membership functions p,f and If' which are associ­ ated with qmagi'(xi, tk) and qmagi'(y, tk), respec­ tively;
8. (e) generate a linguistic rule where Xi are the an­ tecedents, y the consequent , and then the correspon­ dent FR where p,f and If' are the fuzzy sets.

Remark 2. The number of rules generated in correspon­ dence with each QB is not greater than the cardinality of T. Identical rules may happen to be generated as variables may have the same qmag either at different time-points or time-intervals: in such a case we group the equivalent rules and keep only one of them as repre­ sentative of an input-output relation. Qualitative values of the variables of interest which differ only in the value of the output variable are likely generated. Then, it is probable that some conflict rules, i.e. rules which have the same antecedents but a different consequent are pro­ duced. We store such rules and leave the conflicts be resolved in accordance with the degree of rule calculated on the data pairs [Wang, 1994].

Remark 3. The entire range of possible system dynam­ ics is captured by the whole tree of behaviors. Such a tree, as it includes all logically possible outcomes of the given qualitative information, may contain ambigu­ ous results and may be quite large to be efficiently ex­ plored. Such ambiguities, as well as the dimension of the behavior tree, may be significantly reduced by ad­ ditional knowledge, which is not explicitely expressed in the model. Moreover, let us remark that many behaviors in the tree may not present any difference as far as the input-output variables are concerned. Then, behavior aggregation procedures, which aim at taking the signif­ icant distinctions out of the tree, have to be performed to cope with the problem of an efficient generation of a complete and meaningful fuzzy rule base. Whenever reasonable physical assumptions allow us to define equiv­ alent behavior classes, the portion of tree considered is further reduced by keeping one representative for each class. In the extreme case, where all spurious behaviors are not filtered, the performance of FS-QM may get close to that one of a pure black-box FS initialized with the same number of rules.

Given the behavior tree, the algorithm for the gen­ eration of the FRB which describes the overall system dynamics is performed as follows:

1. selection of an individual behavior;
2. generation of the FRB which corresponds to each in­ dividual behavior;

3. union of all the FRB's generated at step 2, and filtering of the equivalent rules.

4 An application of the FS-QM method

We consider the problem of describing the kinetics of Thiamine (vitamin B1) and its phosphoesters in the cells of the intestine tissue of rats, which are known to have human-like Thiamine metabolism. Thiamine is transported in the extracellular fluids in two differ­ ent chemical forms, simple Thiamine ( Th) and Thiamine Mono-Phosphate ( ThMP), and is transformed within the cells through an enzyme-mediated chemical reaction in a higher energy form that is used in the carbohydrates metabolism. The chemical reaction is nonlinear, and modeling through ordinary differential equations is ham­ pered by identification problems. By using FS-QM we build an approximator of the system dynamics, that can be of crucial help in describing syndromes with Thiamine deficiency, like severe liver diseases.

A compartmental model of the Thiamine kinetics in the intestine tissue is shown in Fig. 2 [Rindi et al., 1980] . The variables U1 and U2 represent the amount of Th and ThMP in the plasma, respectively. Th is directly transformed into the Thiamine Pyro-Phosphate ( ThP P). ThMP is firstly transformed into Th, and then from Th to ThPP. Finally, ThPP is dephosphorilated to ThMP. The flow variables (fij, i = 0, .. ,5, j = 1, .. ,5) (chem­ ical reactions) express nonlinear saturable relationships between quantities entering and leaving a compartment. The data set available for the identification of the ap­ proximator is quite rich as we have data on each state variable of the model in Fig. 2. We can completely ex­ press the Thiamine intracellular kinetics by subdividing the overall identification phase into the identification of three approximators as follows:

In the following we will describe in detail how the ap­ proximator of ThPP (5) has been derived.

The Th - ThPP pathway can be modeled through a single QDE. The chemical reactions from Th to ThPP and from ThPP to ThMP are denoted by h (Th) and h (T hP P), respectively. T h acts as input to the sub­ system considered. The qualitative model is described by:

where:

1. Th is a triangular shaped function of time, which represents the input signal. Its quantity space is defined as (0 Th* oo), where Th* is the saturation threshold of JI(Th) .

2. ThPP has quantity space (0 ThPP* oo), where ThPP* is the saturation threshold of h(ThPP) .

3. Both fi(Th) and h(ThPP) are represented in QsiM by the functional constraint s+. From the physiology, we know that h(Th*) > h(ThPP*).

Since the data used for the system identification come from experiments with tracers, the initial value of the variables involved in the simulation is set to 0. The sim­ ulation of the QsiM model produces a tree of 36 quiescent behaviors: 16 of them are filtered out as physiologically inconsistent with the hypothesis h(Th*) > h(ThPP*). Among the remaining behaviors, four of them (Fig. 3) are representative of all the possible behaviors of Th and ThPP. The complete FR8 automatically derived from the qualitative behaviors identifies the number M of fuzzy rules as equal to 15.

5 Results

A simulator of the intracellular Thiamine kinetics has been achieved through two phases: an identification phase, in which the parameters of the derived FS are refined by using 8P algorithm on a first set of data, and a forecasting phase, in which the identified FS has been used as simulator, and its results are matched for vali­ dation against a new data set.

The data used for identification come from an exper­ iment on a group of rats whose intestine tissue was an­ alyzed after an intravenous bolus of 30J.Lg of thiazole­ [214C]Thiamine, with a radioactivity of 1.25 J.LCi, for a period of 240 h, sampled with irregular time intervals.

The performance of the approximator y(J<.) obtained by using FS-QM has been compared with that one iden­ tified by using only the data (denoted by FS-88), as pro­ posed by Wang [Wang, 1994]. The comparison has been performed through the calculation, in both L2 and L00 norms, of the absolute errors between the data and the calculated values.

If the number of 8P loops (nl) is kept low (:S 25), we get absolutely better results of the FS-QM over FS88 both in the identification and forecasting phase. For a greater number of loops, both methods show similar performance (Fig. 4): for nl ---+ oo, the errors calcu­ lated go to zero in both cases with a slight difference in convergence velocity.

In spite of the comparable identification performance, in the forecasting phase y(J<.) performs quite well, whereas the approximator obtained by FS-88 is not able to reproduce the data ofT hP P measured in different ex­ perimental settings. This is not surprising and may be explained by two occurrences: the data are noisy, and the number of samples is smaller than the number of pa­ rameters to be identified. This means that FS-88, which relies only on data, is more likely driven to learn also the noise. Fig. 5 and Fig. 6 compare the results obtained by both methods when applied to simulate the kinetics of T hP P in response to two different input values for T h, which correspond to the values of Th injected on two groups of diabetic rats, one of them under therapy. As it is highlighted by the plot comparison, the performance of the FS-QM approximator is good, while the FS-88 ap­ proximator does not reproduce at all the dynamics of ThPP.

6 Open problems and conclusion

The application of FS-QM to simulate the dynamics of nonlinear systems has given good results which confirm its validity in terms of efficiency and robustness. How­ ever, several problems are still open. First of all, the method requires for a better mathematical formalization which defines its range of validity and applicability. Im­ portant issues which need a thorough study are listed

below:

Membership functions- The choice of the membership functions is a crucial step of both the design and the identification of a FS. Gaussian Membership Functions ( G-MF's) guarantee the Universal Approximation Prop­ erties [Wang, 1994], and are functions of only two param­ eters (mean and variance). Moreover, G-MF's provide for good generalization properties as they ensure com­ pleteness (V;£ E U, J-L(;£) > 0), even when the fuzzy rule base spans only a subset of the Cartesian products of the input space. On the other hand, G-MF's are symmetric and with a maximum value located in a single point: this means that some desirable properties, such as different shapes of the J-Li associated with either a landmark or an interval, cannot be represented. In order to preserve the advantages coming from G-MF's but to improve the ca­ pability of expressing prior knowledge, in the future we will investigate the use of Pseudo-trapezoidal functions [Zeng and Singh, 1996].

Time - The major problem deals with the definition of the mapping of the sampling time set into the qualita­ tive time set, i.e. of the mapping of the measurement grid into the "event" one. This is feasible only if the experiment has been designed so that the data set is in­ formative enough to produce the system dynamics. Such an issue would be facilitated if semi-quantitative infor­ mation on qualitative times would be available.

Hybrid models- The more complete is the a-priori knowl­ edge exploited in modeling, the closer to the solution is the initial guess generated, with a consequent im­ proved efficiency of FS-QM. A semi-quantitative for­ mulation and simulation of the model [Kuipers, 1994; Shen and Leitch, 1993] is hence preferable. Unfortu­ nately, the quantitative information may be insufficient for a semi-quantitative formulation with the mentioned approaches. Therefore, methods for dealing with hybrid models, where different knowledge sources can coexist would be the ideal way to get as much information as possible from the prior knowledge.

Other identification procedures - An interesting strategy to be investigated consists in fixing the parameters x{ and a{ of the vector fl., so that only the parameters YJ are estimated from the data. Since the equation ( 1) is linear in the parameters YJ, it is possible to resort to efficient linear methods, such as ordinary ,least squares. This choice allows us to preserve the structure of the FS initialized on the basis of the a-priori knowledge rep­ resented by the qualitative model, but could prevent from identifying a really "good" approximation of the unknown function. Therefore, possible optimal solutions could be given by two-step identification procedures.