REMOTE AGENT FOR CONFORMANCE TESTING OF TIMED-AUTOMATA Ariel Stulman1, Simon Bloch1,2 , H.G. Mendelbaum2,3 1Univ. Reims, CReSTIC (LICA), Reims, France 2 Jerusalem College of Technology - POB 16031 – Jerusalem, Israel 91160 3 Univ. Paris V, IUT, 143-av. de Versailles, Paris 75016, France Email: {stulman, mendel}@jct.ac.il [email protected]

ABSTRACT The new field of testing of timed-IUT (Implementation Under Test with time constraints) is rapidly evolving, with new and extended test generation methods using timed automata. In all methods, however, it is assumed that the test is conducted locally, with the assumption that there are instant communication times for the transferring of data between the tester and the timed-IUT. In this paper we wish to extend the testing methods to function from remote locations as well; such that we will not assume instant data transfers. We propose a method, which allows the tester to send a testing agent to be added to the timedIUT, using any underlying communication network, although its’ transfer time is unknown. We propose a remote testing architecture with a general algorithm for the testing agent, such that receiving of output by the tester prior to the sending of the next input in the testing sequence is not restricted. An example is given to show the relevance of this method in critical time context. KEY WORDS Timed automata, testing, remote testing, network environment.

1. Introduction Motivated mainly by automata theory, the field of program testing was heavily studied many years ago. Kohavi’s book gives a good exposition of the major results [1]. During the 80’s the topic mostly died down, only to come up again due to its application in the field of communication protocol testing. The abundant and diverse research in the area motivated many algorithms and methods that attempted to optimize the testing procedure (status messages [2], separating family of sequences, distinguishing sequences [3], UIO sequences [4,5], characterizing sequences [6,7,1], and identifying sequences [1]). A survey of the main methods can be found in [8] and in [9]. The International Standards Organization (ISO) quickly realized the need for standardization of conformance testing, and the idea was realized in [10]. Due to the wide use of finite state machines (FSM) as the modeling technique for the systems under test, its inherent flaw was automatically projected into the field of conformance testing. Standard FSMs do not take into account the temporal constraints that may be applicable

within the system; and as such, most methods developed for conformance testing were not applicable to real-time systems. With the proposal of Allur and Dill's 'Timed automata' [11] in 1994, the entire new field of conformance testing for real-time (reactive) systems emerged. A number of papers were published on the topic ([12,13,14], and others). It is easily noticeable, that conformance testing in a remote environment is a non-topic when sequential, untimed systems are in question. The generated test sequence (by any method) must only be transferred to the implementation unit under test (IUT) by some medium (internet, Ethernet, etc.), and the output sent back to the tester. Under such circumstances, it does not really matter if the IUT is local or remote. The transfer time of the sequence is of no importance. Delays by the transferring medium do not influence the test results, and only loss of information can have an impact. With the advancement of network technology, that too is becoming scarce. Such is not the case when we deal with time constraints and time dependence. If the test sequence does not reach the timed-IUT (t-IUT) within the timing constraints required, or if the results are delayed and are not received by the tester within some pre-required time, the t-IUT might be deemed as faulty even though that is not the case (and visa-versa). We wish to propose a method for remote conformance testing of timed-IUT that will overcome the problems that arise due to the use of a remote environment for the test. This paper is organized as follows: In section 2, we introduce all the preliminary knowledge and concepts that pertain to the current discussion. We describe the type of timed automata used, and the testing environment. In section 3, we discuss the main testing methods proposed in the literature. In section 4, we propose a solution to the problem of remote testing and we discuss the advantages and trade-offs of our contribution. Then we point out the further work that is relevant to perform in the field. In section 5, we give an example to show the relevance and importance of our contribution.

2. Preliminaries 2.1 Timed I/O Automata Most conformance testing and test generation techniques that have been previously proposed in the literature were

based on a variant FSM: the famous Mealy machine [15]. This fact is based upon the ability of the Mealy machine to exchange messages with its environment. A deterministic transition is stimulated by input from the environment, and as a consequence the machine can return an output message back to the environment (a reactive machine). Formally: A Mealy machine M is a 6-tuple

S , s 0 , I , O, λ , δ , where: S is a finite set of states. s 0 ∈ S is the initial state of the system. I is a finite set of input events ( I = ι1 ,ι 2 ,...,ι p ).

{

}

O is a finite set of output events ( O = ο1 , ο 2 ,..., ο j ).

{

}

λ is the state transfer-function (λ: S×I → S). δ is the output function (δ: S×I → O). A timed automaton is a Mealy machine with the addition of temporal constraints on the transition function λ. A transition cannot fire, and hence no output or internal state change can occur, if the time values or expressions that restrict the transition are not met. Obviously, a clock or set of clocks must be included within the system to allow for the definition of time. Let a clock constraint, ∆, over a set C of clocks be defined as a Boolean expression of the form x op z, where x ∈ C , op is a classical relational operator (=,≤,≥,>,<,≠), and z is a mathematical expression composed of integer constants and/or clock values and evaluates to an integer. A clock guard, ψ, over C is a conjunction of clock constraints over C: (ψ = ∆ 1 ∧ ∆ 2 ∧ ... ∧ ∆ m ) . Formally: A timed i/o automaton TA is a 7-tuple

S , s 0 , I , O, C , λ , δ , where: S is a finite set of states. s 0 ∈ S is the initial state of the system. I is a finite set of input events ( I = ι1 ,ι 2 ,...,ι p ).

{

}

O is a finite set of output events ( O = ο1 , ο 2 ,..., ο j ).

{

}

C is a finite set of clocks. λ is the state transfer function (λ: S×I×ψ → S). δ is the output function (δ: S×I×ψ → O). Assumption 2.1: For simplicity, we assume that the transfer between states, and hence the output, is instantaneous. That is, if the automaton TA receives ι k ∈ I at instant t, it will also output the corresponding

οb ∈ O

at that exact time. [Justification: In essence, the

output is generated on the same transition. By taking a transition to be infinitely small, we may consider the output as coming out at the same instant.] 2.2 Testing Sequence and t-IUT Output In our remote testing environment, we are unconcerned with the method used for generating the test sequence, just merely with the input sequence itself. Let σ = σ 1σ 2 ...σ n be the test sequence for a timed input/output automata TA, with

ι k ∈ I TA

σ k = ι kτ k

; where:

(the input alphabet accepted by TA), and

τk

is

the timing constraint of the input (the exact time the tester must input ι k after ι k −1 ). It is assumed that

0 <τk < ∞. Let Θ = {ο1 ∨ ε }{ο 2 ∨ ε }...{ο n ∨ ε } be the output generated by the t-IUT in response to the testing sequence σ; where: ok is the output number k and ε is a non-event (a silent response), and {ο k ∨ ε } is the response to the input event ι k ∈ σ k ∈ σ . 2.3 Testing Architecture There are currently various kinds of architectures used for testing. The basic architecture used for testing is shown in Fig. 1 (for some figures see Appendix). It represents the direct link between a tester and an IUT for a sequential un-timed testing. This is the standard architecture that is proposed by [10]. We will demonstrate our solution on the architecture shown in Fig. 2. It is an architecture similar to the ISO standard (Fig. 1), with the addition of the remote environment (internet, Ethernet, etc.) depicted as a cloud. Assumption 2.3: It is assumed that the transfer time from the t-IUT to the first outside location (the closest connection node of the network) is instantaneous. That is, the first external entity is assumed local to the t-IUT, with all the solutions needed for local testing of timed-IUT applying here as well.1 2.4 Controllability and Observability The controllability problem is defined as the task of guaranteeing that ∀σ k ∈ σ the IUT receives ι k −1 1

We must note, that this is not the real case. Time will always elapse between the sending of a message and the time that it is received, no matter how short the distance. We, however, want to demonstrate the effect of the transfer medium on remote testing. Therefore, we make the assumption that is normally made in the context of local testing: instant transfer times between the IUT and the local tester. To justify this assumption, we will note that the message transfer times from remote locations are far greater than their local counterparts. So, when we deal with remote transfer times we may assume local transfer times negligible.

before it receives

ιk ,

and the timing constraint

τk

is

realized. The observability problem is defined as the task of guaranteeing that ∀ο k ∈ Θ , the tester receives

ο k −1

before

οk

is received. The tester must also have the

ability to distinguish between the outputs of the t-IUT; to know which output was stimulated by which input event (See Fig. 3). In addition, we must check the timing of the outputs to see if the constraints on the state transfers were Observability problem in remote env.

ιk

ιk+1

Tester Time IUT

ε response?

ob

Figure 3: Observability Problem in remote environment (is ob the response to ιk or was there a silent response (ε) to ιk, and ob is the response to ιk+1 ?)

met. There are many factors that might hinder the controllability and observability of the test system. Some, such as the loss of information or reordering of data, influence the data portion of the messages ( ι k and ο k );

output, one at a time, and decide whether to continue the test, what should be the next input, etc. The additional overhead incurred by the underlying network environment is of no real importance to the tester (assuming it works properly). 3.2 Testing of Timed-IUT If we attempt, however, to test a timed-IUT directly, we may have to revert back to the theoretical testing scheme. It may be impossible to use an adaptive test due to the timing nature of the machine. We must again pre-generate a test sequence σ, test the timed-IUT (t-IUT), and compare the outputs to the ones expected. When such is the case, the difference between a local, instantaneous testing environment and a remote, time consuming environment can be crucial. 3.2.1 Remote Testing Even if we assume the existence of a testing scheme for the testing of a t-IUT using timed automata, when we extend the solution to a remote testing environment, we must consider the influence of the underlying network on the timing of the I/O messages. The delay, loss of information, and other miscellaneous factors must be corrected or overcome. Solution in [16] and [17]

ok

others, such as the delay of messages, influence the timing of the messages ( τ k ), and therefore the test of the TA. All

Tester

of these errors caused by the use of a remote testing environment must be taken into account and corrected or overcome.

2

The IUT’s next input depends on the result of the previous output. At each step the IUT’s output is examined by the tester in order to determine what the next input in the test sequence should be. In essence, an adaptive test is not a test sequence but rather a decision tree. 3 The testing sequence is predetermined before testing commences.

ιk+1 tc Time

IUT

ιk

3. Discussion on testing 3.1 Local and Remote Testing of un-Timed IUT It is well know that conformance testing is theoretically composed of three stages. The first stage is concerned with the generation of a test sequence. The second stage is concerned with the conduction of the test. The third and last stage, is the comparison of the outputs from the IUT to the ones expected based on the specifications (SPEC). If all outputs are as expected, we can accept the IUT as conformant to its specification. The use of a remote environment for the execution of such a test does not impose any changes on the discussion above. The test can be adaptive2 or static3, and in either case we can “take our time”. We can examine each

tc for ok=ε

tc

Figure 4: Restrictions placed on the timing of messages in [16] and [17], in order to solve the observability problem

In [16] and [17] equations that must be satisfied when using a transfer medium are provided, but a number of assumptions are made about the testing architecture that deem the equations irrelevant for our t-IUT. For example (see Fig. 4), in order to solve the observability problem, it is assumed that an input ι k +1 ∈ I cannot be sent until the output

οk ∈ O

for

ιk ∈ I

is received, or some

maximum time limit for the receiving of output is exceeded (for the case where there is no output in response to ι k ∈ I – a silent response (ε)). The timing constraints ( τ k ) in the model refer solely to the time between the receiving of transmission of

οk ∈ O

by the t-IUT and

back to the tester, and the time

between the receiving of transmission of

ιk ∈ I

ι k +1 ∈ I

οk ∈ O

by the tester and the

to the t-IUT. Under such

assumptions, equations are formulated that allow the

proposed solution (semi-local testing)

ok

tc

instant transfer time

input sequence is entered correctly, ∀σ k immediately following

Agent Time

proposed algorithm is shown in Fig. 7. procedure testing_agent_layer begin receive σ; reset iut_clock; reset agent_clock; while (i
t-IUT

ιk

ιk+1

Figure 6: Semi-local testing assuming instant transfer times between local testing agent and t-IUT (constraints (tc) placed only on timing between inputs)

while(

4.1 Remote Testing using a Local Testing Agent It should be noted that there is an inherent trait in the theoretical testing scheme that can be exploited so as to overcome the problem at hand. The three stages, due to their distinct nature, do not all have to be executed at or from the same remote location. We can generate a test sequence and check the results at a remote location, while the test can be conducted locally. After the generation of the testing sequence σ, the remote tester can send it to the nearest external location (closest network node) to the t-IUT. From that location, it was assumed (assumption 2.3) that the transfer time to the t-IUT is instantaneous. The test can now be conducted locally, logging locally the outputs from the t-IUT together with the time of their reception. At the end of the test, the logged outputs are sent back to the remote tester for examination and comparison to the expected outputs. In essence, we propose to add to the t-IUT testing architecture a layer that will act as a local agent so that the test can execute correctly despite the fact that the tester is remote. The new architecture can be seen in Fig. 5 (see Appendix). Under this new testing architecture, we can relax assumption 2.3; it is no longer necessary to assume instantaneous transfer time between the t-IUT and the first network node. Rather, a more realistic instantaneous transfer time between the t-IUT and the local testing agent layer added to the architecture (see Fig. 6). 4.2 Building the Testing Agent Layer The additional testing layer to the t-IUT is a small agent sent by the remote tester to help with the timing of the test. The purpose of the testing layer is to guarantee the

held. A

4

// based on OSI //resets t-IUT’s clock // resets testing agent’s clock // |σ| = n

(τ k ∈σ k ) − (τ k−1 ∈σ k−1 ) ≠ agent_clock)

wait;

tester to accept or reject the t-IUT. Also, inconclusive results can occur if some of the equations are unsatisfied. We cannot, however, send input events to the t-IUT without waiting for the response (consequent outputs) of the previous input event, a very stringent constraint when dealing with remote environment and timed-IUT. If we were able to eliminate the transfer time, or if we were to know the exact, constant transfer time, this restrictive constraint would immediately disappear and the equations can again be used.

4. Proposed solution

σ k −1 , with the timing τ k

entered

input( ι k ∈ σ k ); if (output)

ο k ∈ Θ = output else log( ο k ∈ Θ = ε ); log(

reset (agent_clock); end while; send(Θ); end procedure;

);

// on σ inputs // based on OSI

Figure 7: Local agent's algorithm

By using the OSI model as the basis for the data communication of the test sequence from the remote tester to the testing layer, the order of σ is held and all possible lost data recovered (a requirement that is built into OSI). Since the actual test has not yet commenced, we may allow as much time as needed to make sure that the transfer was correctly accomplished. From its local location (for which instant transfer time to the t-IUT was assumed – assumption 2.3), the timing constraint τ k can be easily guaranteed; thus, solving the controllability problem. By assumption 2.3 and assumption 2.1 the testing layer can easily distinguish between an output ο k and an empty output ε. If in response to

ι k ∈ σ k at

instant

τk

output is instantaneously detected, it can be logged as ο k ∈ Θ . If no output is detected at time τ k , the testing layer will log

οk = ε

as the output to

ι k ∈ σ k . Using

such a technique, the testing layer easily constructs Θ correctly. Once the test is completed, the testing layer must send Θ back to the remote tester while overcoming the second half of the observability problem. Again, the OSI model for computer communication over any medium solves the problem (taking all the necessary time). No loss of data or reordering of the outputs ο k ∈ Θ will occur. Once more, we may take the time needed to guarantee the safe transfer of Θ since the test is actually over. Now, at the 4

In the proposed algorithm it was assumed that there is only one clock in the t-IUT, iut_clock. This assumption can easily be relaxed without much change to the underlying algorithm.

testers’ remote location, we can compare the results from the t-IUT to make sure that ∀ο k ∈ Θ corresponds to the output expected.

5. Example To demonstrate the relevance of our solution, we will use the diagram in Fig. 8. It is a diagram of some portion of the timed automaton that is used to test the t-IUT at hand. The transition input event and timing constraints on the transitions of the timed automaton are of the form σ k ψ \ ο k c ; where: σ k ∈σ ,

[ ]

[]

ψ = (∆1 ∧ ∆ 2 ∧ ... ∧ ∆ m ) , ο k ∈ Θ

and c is a clock

assignment such as reset, setting the clock to a value (assuming there is some clock assignment needed). Of course, we can omit any portion of the form, causing the transition to disregard that missing section. Portion of a Timed Automaton σm [lc
q3

q2 σj \ ε [lc=0]

σm [lc>=X] \ ε

q4

Figure 8: A timed-automata-branch that is dependent upon time

It is obvious to see, that this section of the timed automaton depicts a case of a branch that is dependant upon the timing of the input event. We begin at state q1, where an input of σj is accepted regardless of the timing. When such an event is entered, the transition will fire. The automaton advances to state q2, returning a non-event (ε) to the environment and resetting the local clock (lc) to 0. Now, the automaton is at a state q2, i.e. it is ready to accept σm. The branch, however, depends on the timing of the input event. If it is entered when lc is less than some constant, X, we will advance to q3. Otherwise the automata will advance to q4. In essence, this is the case where two events (σj and σm) must occur within some time frame for the automaton to advance through a specific route. In [16] and [17] a restriction on the sending of consecutive input events was placed in order to overcome the observability problem (see section 3.2.1). Under that specific constraint, the above t-IUT cannot be tested if the transfer time (TT) between the t-IUT and the tester is greater than half of the constant X.5 5

The inequality is based upon the fact that we must wait for the result from σj before we can send σm. If we require X/2 time units for the TT of the result of σj and the same for the TT of σm, the sequential execution of both will inevitably take X. Hence, if TT > X/2, it is impossible for the t-IUT to receive σm by time constrained by X.

In our model, however, the test is conducted using some local agent (a testing layer). From the agent to the tIUT the transfer times were assumed instantaneous (assumption 2.3); hence, we are in full control to choose when to enter the second of two consecutive input events. We can test the above t-IUT even one wish to use a remote testing architecture for the test.

6. Conclusion We have shown in this paper the major difficulties of conformance testing of t-IUTs when using a remote environment. Our main contribution is the proposed solution for conducting such a test while overcoming the transfer time of the transport medium. We also proposed a model that is not constrained by the receiving of output for sending of the input in consecutive events. This allows us to use existing equations (for example [16] and [17]) to determine conformance to a SPEC. In addition, an algorithm is provided for the local testing agent suggested.

7. References [1] Z. Kohavi, Switching and finite automata theory (2nd ed, McGraw-Hill, 1978). [2] W. Y. L. Chan, S.T. Voung, & M.R. Ito, An improved protocol test generation procedure based on UIO’s. Proc SIGCOM, 1992, 283278. [3] G. Gonec, A method for the desing of fault detection experiment, IEEE Trans. on Computers, C-19, 1980, 551-558. [4] K.K. Sabnani, A.T. Dahbura, A protocol test generation procedure, Computer Networks and ISDN Systems, 15, 1998, 285-297. [5] A.V. Aho, A.T. Dahbura, D. Lee, & M.U. Uyar, An optimization technique for protocol conformance test generation based on UIO sequences and rural Chinese postman tours, IEEE Trans. Commun.ications, 39, 1991, 1604-1615. [6] T.S. Chow, Testing software design modeled by finite state machines, IEEE Trans. Software Engineering, SE-4(3), 1978, 178-187. [7] G. Luo, G. von Bochmann, & A.F. Petrenko, Test selection based on communicating nondeterministic finite state machines using a generalized Wp-method, IEEE Trans. Software Engineering, 20(2), 1994, 149-162. [8] D. Lee, M. Yannakakis, Principles and methods of testing finite state machines – a survey, Proc of the IEEE, 84(8), 1996, 10901126. [9] E. Brinksma, L. Heerink, & J. Tretmans, Developments in testing transition systems. International Workshop on Testing Communicating Systems X, Chapman & Hall, 1997, 143-166.

[10] ISO, Information technology – OSI. ISO international standard 9646: Conformance testing methodology and framework, Geneva, Switzerland, 1991. [11] R. Alur, D. Dill, A theory of timed automata, Theoretical Computer Science, 126, 1994, 183235. [12] S. Bloch, H. Fouchal, E. Petitjean, & S. Salva, Some issues on testing real-time systems, International Journal of Computers and Information Science, 2(4), 2001, 230. [13] S. Salva, H. Fouchal, & S. Bloch, Metrics for timed systems testing. Proc. 4th OPODIS International Conference on Distributed Systems, Paris, France, 2000, 177-200. [14] K. Naik, B. Sarikaya, Protocol conformance test case verification using timed – transition. Proc. 14th International Symposium on

Protocol Specification, Testing and Verification, Vancouver, Canada, 1994. [15] J.E. Hopcroft, J.D. Ullmann, Intoduction to automata theory, languages, and computation (Reading, Massachusetts: Addison-Wesley, 1979). [16] A. Khoumsi, Testing distributed real-time reactive systems using a centralized test architecture. IEEE North Atlantic Test Workshop (NATW), Gloucester, Massachusetts, 2001. [17] A. Khoumsi, Testing distributed real-time systems: an efficient method which ensures controllability and optimizes observability. Proc. Conference on Real-Time Computing Systems and Applications (RTCSA), Tokyo, Japan, 2002.

Appendix (Figures):

Tester

Tester

IUT

Figure 1: Standard Testing Architecture

Network Testing Layer

Tester IUT Network Figure 5: Remote Testing w/ Testing-Agent Layer

IUT Figure 2: Remote Sequential Testing