Application of Formal Methods for Establishing Regulatory Requirements for Safety-Critical Software of Real-Time Control Systems Sergiy A. Vilkomir

Jonathan P. Bowen

South Bank University (SBU) Centre for Applied Formal Methods (CAFM) School of Computing, Information Systems and Mathematics (SCISM) 103 Borough Road, London SE1 0AA, UK Email: {vilkoms, bowenjp}@sbu.ac.uk URL: http://www.cafm.sbu.ac.uk/ Abstract Formal methods are usually used for computer system specification, production and verification. In this paper, a new direction for the use of formal methods is proposed, namely formalization of the regulatory requirements for software of safety-critical real-time control systems. Formal regulatory requirements, as opposed to formal specifications for a concrete system, have a generic nature, are applied for a wide range of safety-critical control systems, and are the basis for a licensing process. The use of formal regulatory requirements could help to eliminate ambiguity or misunderstanding of informal definitions, to allow rigorous assessment of satisfaction with respect to requirements and finally to increase the safety level of a system. As a formal method for describing the regulatory requirements, the Z notation is proposed. To demonstrate the proposed approach, requirements for protection against common mode software failures and requirements for protection against unauthorized access are considered. Z schemas for these requirements are presented.

1

Introduction

A set of formal methods is a methodological framework for system specification, production and verification. These methods provide means of the development and assessment of a system or parts of a system (e.g., software), based on principles such as logical reasoning, set theory, abstract algebra or graph theory, that allow mathematical manipulation and proof of properties. Formal methods are especially recommended for safety-critical applications [6], where high integrity is important. The designer of the real-time control system and software has the main responsibility for ensuring safety. The application of formal methods to safety-critical systems from the point of view of designers, i.e., for system or software development, is considered in plenty of scientific and technical publications. In many countries, in areas such as air transport and nuclear energetics, there are also special state regulatory bodies (authorities) whose basic tasks are independent control of safety and licensing of activity connected with safety. Concerning computer control systems, there are two main tasks of the regulatory body: 1

• establishment of regulatory requirements for systems and software; i.e., requirements (criteria, norms, rules) important for safety; • an assessment of the system and software during licensing (regulatory assessment). The purpose of this assessment is to be convinced that the system and software answer established regulatory requirements. These tasks are complex, labour-consuming, and have a specific character. It is expedient to use the special methods and tools for the decision of these tasks. The application of formal methods during licensing is considered in a lesser degree than for software development and is not widely put into practice at present. Nevertheless, the application of formal methods in this area can be useful and successful. When formal methods were used during licensing, it was mainly for systems or software assessment, i.e., for solving the second of the tasks above. Thus, with regard to using formal methods, application of software tools for static and dynamic analysis could be considered. Examples of using such tools for the regulatory assessment during licensing include using MALPAS [34] for the French nuclear plants protection system SPIN software assessment [14] and LDRA Testbed and VALIDATOR tools for the independent assessment of the Teleperm XS software [23]. The example of manual formal approach is using “the function tables” for safety assessment of software for the shutdown systems of the Darlington Nuclear Plant, Canada [28]. Various approaches to using formal methods in safety analysis, which can be useful for regulatory assessment of real-time safety-critical systems, are considered in [2, 9, 12, 16, 26, 30]. In this paper, a new direction is proposed, namely the formalization of the regulatory requirements for software of real-time control systems. This involves the use of formal methods for establishing these regulatory requirements (the first of the mentioned tasks above). Formal regulatory requirements, as opposed to formal specifications for a concrete system, have a generic nature and are applied for a wide range of safety-critical control systems and software. The use of formal regulatory requirements could help to eliminate various understanding of problems or ambiguity of informal definitions, to allow rigorous assessment of satisfaction to requirements and finally to increase the safety level of a system. This approach is described below in Section 2.

2

Formalization of the regulatory requirements for software

2.1

General Approach

The regulatory requirements can be determined in: • normative documents of the regulatory authority; • national standards; • international standards (IEC, ISO, etc.); • recommendations of international organizations (IAEA, etc.). In any case, the regulatory authority should approve using these requirements for a branch (in the framework of its responsibility). It is possible to divide the regulatory requirements for software of real-time control system on the various groups, for example [37]:

2

• requirements for the structure and elements of software; • requirements for the diagnostics; • requirements for protection against failures and distortions; • requirements for software development procedures; • requirements for software testing. The regulatory requirements are usually the general requirements for software of safety-critical systems, irrespective of their functionality, for example, for all safetycritical control systems at nuclear power plants. These systems have various purposes, have various levels of completeness and sizes, and are developed various designers with using various methods and techniques. So, the requirements should possess contradictory characteristics. On the one hand, they should be general enough to leave a designer a free choice of technological techniques, i.e., they should not be dependent on methods of development. On the other hand, the requirements should be specific enough to show a designer what should be done and how it should be done to satisfy the requirements. It is not always easy to formulate the requirements in such manner. Thus, some informal requirements (formulated in natural language) have the following shortcomings: • they are too general and do not contain instructions on how it is possible to satisfy and to check these requirements; • they are not accurate enough, allow various possible interpretations and require additional explanation. To avoid these situations, we propose to use formalized regulatory requirements. The formalized requirements (i.e., elaborated by using a formal mathematically-based notation) can help eliminate above mentioned shortcomings. The formal definitions are more specific and precise, and ensure common understanding of the requirement contents both between experts within a regulatory authority and between experts and designers. The structure of a formal definition also can facilitate searching the way to prove the realization of this requirement. Another benefit of using formal requirements is the possibility of formalizing and automating the regulatory assessment process. The use of formal requirements facilitates a special software tools application both for support of whole licensing process [36] and for an individual requirement assessment. Sometimes problems with informal requirements arise when the system’s designer from one country elaborate a safety-critical control system for use in another country. The system should satisfy the regulatory requirements of the country where this system will be in operation. However, the designer may be not familiar with these requirements in detail. There could be different interpretation of the same requirements in various counties, even different meaning of the same terms in various languages. In this case, the formal requirements can be used as an “international language” and facilitate mutual coordination between a regulatory authority and a foreign system’s designer. Obviously, the understanding of formal definitions of requirements can be more difficult than definitions in natural language. The use of formal definitions can require a special training to attain a sufficient professional standard. Unfortunately, there is no one common and universally recognized formal method, which is followed by the majority of software developers. That is why we propose to use formal requirements 3

not instead but in parallel with the requirements in natural language. For example, within a normative document of software requirements, the formal definitions can form a recommended supplement and be used only for restricted cases involving the most important and complicated requirements. The formal definitions of requirements could be a good base for discussion between experts of a regulatory authority and a software designer prior to development beginning. They could help to reach the common understanding of further licensing process, contents of regulatory requirements for this particular system (software), and the volume of documentation, which should be submitted for the regulatory review. The selection of a relevant formal method is a separate problem. Generally, many of well-known methods (languages, notations) can be used one way or another for formalization of the regulatory requirements for software, e.g., formal methods as VDM [24, 40] and B [1, 41] or semi-formal methods as UML [3, 11], Statecharts [13, 17] and its variety Safecharts [27]. In this paper, we propose to formalize requirements in the Z notation. The choice of the Z notation is explained by the following reasons: • the main purpose of Z [31] is for specifying and designing software so Z is suitable for describing both general and detailed requirements for software; • there are many examples of substantial and successful practical applications of the Z notation to safety-critical areas [5, 15]; • a lot of textbooks on Z are now available (e.g., [4, 20, 21, 25]); • Z is one of the most widely taught formal methods [7]; • the community of Z users is well organized; Z user meetings are carried out on regular base [8]. Formal regulatory requirements can be applied not only directly to software but also to the process of software development. Thus, one of the most important stages of regulatory assessment is the assessment of a software testing process [38]. In this case, co-called testing criteria can be used as regulatory requirements. For example, according the DO-178B standard [29], the testing process must correspond to Modified Condition/Decision Coverage Criterion (MC/DC). Software should satisfy this standard in order to be certified by the FAA (Federal Aviation Administration, USA), so this requirement is regulatory. The formal definition of MC/DC in Z notation is given in our paper [39], where Z schemas for another primary control-flow testing criteria are also introduced. This formal definition is strict and eliminates inaccuracy, which is intrinsic in definitions using natural language. Two examples of using the Z notation for formalization of regulatory requirements, typical for real-time control systems, are considered below, namely requirements for protection against common mode failures (section 2.2) and requirements for protection against unauthorized access (section 2.3).

2.2

Requirements for protection against common mode failures

Protection against failures (fault-tolerance) is one important method that can help ensure a high level of reliability of a real-time system. In order to guarantee the tolerance against failures of hardware, multi-channel redundant real-time systems are used, for example, three-channel systems with “2 from 3” logical voting of signals. However, this approach does not guarantee the due protection against software failures. In the case of a software error, identical programs in different channels can 4

have a simultaneous failure. It is possible to ameliorate software failures with the help of software itself, so the following regulatory requirement is very important for safetyrelated real-time systems: software should carry out the protection against common mode failures, including failures because of software errors. For protection against common mode software failures, the following approaches are possible [37]: • the use of two different computer systems (primary and diverse) for the fulfillment of the same functions, but constructed on different hardware and software. For example, primary and diverse protection systems are parts of NPP Temelin (Czech Republic) design modification project; • the use of various software implementations in various channels of the system for the fulfillment of the same functions; • the use of more than one criteria of system actuation for each situation, when the action of the system is necessary (functional redundancy). There are different opinions of the expediency of these approaches because each one has own advantages and disadvantages. The use of the latter approach (functional redundancy) is possible for real-time systems, which work in a waiting mode, i.e., when the actuation of the system is required in special (emergency) situations. An example of such system is a nuclear reactor protection (shutdown) system. The use of functional redundancy means that each emergency situation (the initial event for the system actuation) should be found out in at least two ways based on various technological parameters. For example, if the first way is based on measuring of temperature (main criterion), the second way could be based on measuring of pressure (additional criterion). In this case, if the system actuation did not take place due to errors in the part of the software that realized the main criterion of an actuation, then the system should operate owing to realization of additional criterion. Using this requirement for real-time system software in practical regulatory assessment requires a more precise definition. Below we consider how this requirement can be formally described using the Z notation. The following given set contains names of all possible input parameters of a system.

[IDENTIFIER] It is necessary to stress that this set is not primordially settled like a set of all input parameters from a system specification. We consider this set also contains names of all potential parameters, i.e., if it is necessary to add a new input parameter during system development, the name of this parameter is in the set IDENTIFIER. The following definitions are also used in Z schemas where R denotes the set of real numbers (real numbers definitions in Z are addressed in [35]):

R1 == {x : R | x ≥ 0} Time == R1 Command ::= wait | act

The following schema1 describes the notion of an input parameter of a system. 1

All schemas in this paper were checked using the ZTC type-checker package [22].

5

parameter ident : IDENTIFIER areaparam : P1 R valueparam : Time → R ran valueparam = areaparam

Each input parameter contains the name ident down from IDENTIFIER. For any point in time t, the parameter has the value valueparam(t). The subset of real numbers areaparam specifies a range of the input parameter, i.e., for all t valueparam(t) is a member of areaparam. The following schema determines contents of a criterion for a system actuation.

criterion seqparam : seq parameter num : N1 statecrit : Time → → Command areacrit : P1 (seq R) #seqparam = num ∀ point : areacrit • #point = num ∀ point : areacrit; i : 1 . . num • point(i ) ∈ (seqparam(i )).areaparam ∀ t0 : Time • statecrit(t0 ) = act ⇔ {i : 1 . . num • i 7→ (seqparam(i )).valueparam(t0 )} ∈ areacrit

Each criterion is based on a set of input parameters (the sequence seqparam); the natural number num gives the number of these parameters. For any point in time, the criterion has one from two possible values of elements of the set Command . The value wait means that no actions are necessary, i.e., the situation is not emergency. The value act means occurrence of an emergency event when the system actuation is required. The state of the criterion is depended from a combination of input parameters values. The set areacrit, which is the subset of the Cartesian product of num sets of real numbers, determines the area of combinations of input parameters where the system actuation is required. For the most part of time, the state of the criterion is wait. On the other hand, according system assignation, the situations for system actuation always exist, i.e., the function statecrit is generally surjective. The following schema determines contents of considered regulatory requirement for the protection against common mode software failures. The set setmaincrit contains the main criteria of system actuation, which determining the algorithm of system operation. The functional redundancy is realized by setting the additional criteria (the set setaddcrit). In this connection, the correspondence between the main and additional criteria should be determined (the function addcrit). The main requirement for the functional redundancy realization consists of independence between any main criterion and an adequate additional criterion. It means that these criteria should be based on different input parameters, i.e., for any main criterion maincrit the sets maincrit.seqparam and (addcrit(maincrit)).seqparam should be disjoint. 6

FunctionalRedundancy setmaincrit, setaddcrit : P1 criterion delay : criterion → 7 7 Time addcrit : criterion → 7 7 criterion #setaddcrit ≤ #setmaincrit dom delay = domaddcrit = setmaincrit ran addcrit = setaddcrit ∀ maincrit : criterion | maincrit ∈ setmaincrit • maincrit.seqparam ∩ (addcrit(maincrit)).seqparam = ∅ ∀ maincrit : criterion; t0 : Time | (maincrit ∈ setmaincrit) ∧ (maincrit.statecrit(t0 ) = act) ∧ (∀ t : Time | t < t0 • maincrit.statecrit(t) = wait) • ∃ t1 : Time | t1 ≥ t0 • (addcrit(maincrit)).statecrit(t1 ) = act ∧ (t1 − t0 ) ≤ delay(maincrit)

For functional redundancy of real-time systems, the factor of time is very important. The problem is that the real-time system actuation according to the additional criterion could have a response lag after initiation of an emergency situation. When this response lag is admissible and when it is not depends on the specific emergency situation and the specific real-time system. So, during the functional redundancy realization, the admissible time delay in the system actuation delay(maincrit) should be established for each main criterion. The value of this delay can variate from fractions of a second to sizeable time intervals. The requirement for the functional redundancy realization is that the gap between time t0 of the emergency situation appearance according the main criterion and time t1 of the system actuation according the additional criterion should be less than previously determined delay(maincrit) volume. The presented Z schemas can be practically used for carrying out of the regulatory assessment. In this connection, it is necessary to concentrate attention on validity of the admissible time delays for main criteria and expected time delays for additional criteria.

2.3

Requirements for protection against unauthorized access

The requirements for protection against unauthorized access with reference to software have specific features for each type of computer system. For example, for bank systems and databases the most important is the protection against deliberate attempts of unauthorized access. For real-time control systems, this task is usually achieved by administrative and organizational measures and only special personnel operate such systems. So, the task of real-time control system software is protection mainly against unpremeditated unauthorized actions of personnel. A broad range of specialists (programmers, technologists, system engineers, humanoperators) takes part in maintenance of a real-time control system. An unpremeditated fault during changing of a program code or operate and alarm setting can have grave consequences from safety point of view. Therefore, it is very important to differentiate responsibilities of personnel according administrative functions and duty regulations. Software protection against unauthorized access should allow every member of staff to 7

carry out only a limited set of actions during maintenance of a system in conformity with the personnel duties. This approach is not directly connected with a time factor but it reflects the specific features of maintenance of real-time control systems. It is expedient to reflect this part of a problem in the regulatory requirements for software of such systems. However, many normative documents and standards include this requirement only in a very general form. For example, in IEEE Std 603-1998 Criteria for Safety Systems for Nuclear Power Generating Stations [18] the requirement for control of access is formulated in only two sentences: “The design shall permit the administrative control of access to safety system equipment. These administrative controls shall be supported by provisions within the safety systems, by provision in the generating station design, or by a combination thereof.” IEEE Std 7-4.3.2-1993 Criteria for Digital Computers in Safety Systems of Nuclear Power Generating Stations [19], which specifies additional computer specific requirements, indicates that no requirements beyond IEEE Std 603 are necessary for control of access. But such general requirements may not be enough constructive for using during regulatory assessment. On the other hand, detailed explanations in natural language are not always possible in standards and are not always readable and well-defined. So, it is interesting to consider the possibility of using of formal regulatory requirements for protection against unauthorized access. Formal methods have been used for the specification of access control of various systems. For example, the Z notation was used in the development of formal models for an access control systems for the ADMIRAL project [32]. More recently, VDM and PVS were used in the analysis of the requirements for an access control system, which is part of a comprehensive security system [10]. For the Mondex electronic cash system and the Multos secure smartcard operating system, the Z notation was used to specify the high-level abstract formal specification of the desired security properties, to specify the lower level design, and to provide formal proof of correspondence between the two levels [33]. Formal specifications for recent above-mentioned system have a large size, reflect specific features, and obviously could not be considered as regulatory requirements. Below we examine how the discussed examples in this subsection differentiate the responsibility of personnel and, accordingly, differentiate the possibility of access, and how a generalization of this can be formalized in the Z notation as a regulatory requirement. The following given sets are used to establish this requirement:

[ACTION , NAME , LEVEL] The first set is a set of symbols for describing names of personnel and levels of access. A set ACTION is a set of various actions, which are available for personnel during operation and maintenance of a system, for example, information reading or scan, operate or alarm setting change, technological algorithms modernization, program code replacement, etc. Details of the contents of ACTION should be considered for each specific system and is not a subject of the regulatory requirements. The following schema determines the requirement, where the set staff is a list of operating and technical personnel and the set levels describes different levels of access. Differentiation of allowed access is described by two functions. The stafflevel function determines the level of access for each member of personnel (it is possible, of course, that several members have the same level of access). The actions function determines the contents (available actions) for each level. Usually, the contents of different levels are different, i.e., the actions function is an injection. 8

access staff : P1 NAME levels : P1 LEVEL stafflevel : NAME → 7 7 LEVEL actions : LEVEL  7 7 P ACTION dom stafflevel = staff ran stafflevel = levels dom actions = levels

In spite of its simplicity, this schema is a constructive requirement, i.e., also determines how this requirement can be checked and confirmed. Thus, to confirm realization of this requirement, the designer of specific real-time control systems should submit to the regulatory authority the following information: • possible actions during operation and maintenance of the system; • levels of access with the subset of actions for each level; • positions of personnel with the level of access for each position. The given schema does not cover all aspects of the requirements for access control. For example, the requirements for the identification of personnel (passwords) should be assigned. Here we show the way in which the formal requirements, while maintaining the necessary commonality, can be constructive and easily checked during regulatory assessment.

3

Conclusions and Future Work

The subject of this paper is a proposal for a new direction in the use of formal methods, namely formalization of the regulatory requirements for software of safety-critical realtime control systems. Formal regulatory requirements can be used as a basis for a licensing process when used in parallel with the requirements in natural language. The Z notation [4] is proposed for the formalization of regulatory requirements. Z schemas have been elaborated for regulatory requirements typical for real-time control systems, namely requirements for protection against common mode software failures and requirements for protection against unauthorized access. These strict formal definitions can be used practically for aiding regulatory assessment. The proposed schemas are constructive requirements, i.e., they determine how these requirements can be checked and confirmed. Future work in this area will be directed at selecting requirements for which formalization would be the most effective, producing a set of formal definitions of these requirements, and consideration of the practical application of the elaborated approach. Another direction for future work could be the development of assessment methods to checking conformance to formal requirements.

References [1] Abrial, J.-R. The B-Book. Cambridge University Press, 1996.

9

[2] Atchison, B., Lindsay, P., Tombs, D. A Case Study in Software Safety Assurance Using Formal Methods. Technical Report No. 99-31, Software Verification Research Centre, School of Information Technology, The University of Queensland, Australia, September 1999. [3] Booch, G., Jacobson, I., Rumbaugh, J. The Unified Modeling Language User Guide. The Addison-Wesley Object Technology Series. Addison-Wesley, 1998. [4] Bowen, J. P. Formal Specification and Documentation Using Z: A Case Study Approach. International Thomson Computer Press, 1996. [5] Bowen, J. P., Hinchey M. G. The Use of Industrial-Strength Formal Methods. Proceedings of 21st International Computer Software and Application Conference (COMPSAC’97), Washington D. C., USA, 13–15 August 1997, pp. 332–337. [6] Bowen, J. P. The Ethics of Safety-Critical Systems. Communications of the ACM, Vol. 43, No. 4, April 2000, pp. 91–97. [7] Bowen, J. P. Experience Teaching Z with Tool and Web Support. Technical Report SBU-CISM-00-30, SCISM, South Bank University, London, UK, 2000. [8] Bowen, J. P., Dunne, S., Galloway, A., King, S. (Eds.). ZB 2000: Formal Specification and Development in Z and B. First International Conference of B and Z Users, York, UK, August 29 – September 2, 2000. Proceedings. Lecture notes in computer science, Vol. 1878, Springer, 2000 [9] De Lemos, R., Saeed, A., Anderson, T. Analyzing Safety Requirements for ProcessControl Systems. IEEE Software, Vol. 12, No. 3, May 1995, pp. 42–53. [10] Droschl, G. Analyzing the Requirements of an Access Control Using VDMTool and PVS (abstract). In J. M. Wing, J. Woodcock, and J. Davies, editors, Proceeding of FM’99 – Formal Methods, World Congress on Formal Methods in the Development of Computing Systems, Toulouse, France, September 1999. Lecture notes in computer science, Vol. 1709, page 1870, Springer, 1999. [11] Gomaa, H. Designing Concurrent, Distributed, and Real-Time Applications with UML. Addison-Wesley, 2000. [12] Hansen, K., Ravn, A., Stavridou, V. From Safety Analysis to Software Requirements. IEEE Transactions on Software Engineering, Vol. 24, No. 7, July 1998, pp. 573–584. [13] Harel, D., Politi, M. Modeling Reactive Systems with Statecharts: the Statemate Approach. McGraw-Hill, 1998. [14] Henry, J. Y., Soubies, B., Le Meur, M., Boulc’h, J. Evaluation Methods for the I and C Safety Softwares of Nuclear Power Plants. International Atomic Energy Agency Technical Committee Meeting on Safety Assessment of Computerized Control and Protection systems. Ref: J4-TC-698.2, 12–16 October 1992, IAEA Headquarters, Vienna, Austria. [15] Hinchey, M., G., Bowen, J. P., editors. Industrial-Strength Formal Methods in Practice. Springer, 1999.

10

[16] H¨orl, J., Aichernig, B. Validating Voice Communication Requirements Using Lightweight Formal Methods. IEEE Software, Vol. 17, No. 3, May/June 2000, pp. 21–27. [17] Horrocks, I. Constructing the User Interface with Statecharts. Addison-Wesley, 1999. [18] IEEE Std 603-1998. IEEE Standard Criteria for Safety Systems for Nuclear Power Generating Stations. 1998. [19] IEEE Std 7-4.3.2-1993. IEEE Standard Criteria for Digital Computers in Safety Systems of Nuclear Power Generating Stations. 1993. [20] Imperato, M. An Introduction to Z. Chartwell-Bratt, 1991. [21] Jacky, J. The Way of Z: Practical Programming with Formal Methods. Cambridge University Press, 1997. [22] Jia, X. ZTC: A Type Checker for Z Notation. User’s Guide. Version 2.03, August 1998. Division of Software Engineering, School of Computer Science, Telecommunication, and Information Systems, DePaul University, USA, 1998. [23] Linder, A., Miedl, H. Methodology and Tools for Independent Verification and Validation of Computerized I & C Systems Important to Safety. In: IAEAIWG-NPPCI-98/1 Working Material. Computerized Reactor Protection and Safety Related Systems in Nuclear Power Plants. Proceedings of a Specialists’ Meeting, 27–29 October 1997, Budapest, Hungary, pp. 127–138. Reproduced by the IAEA, Vienna, Austria, 1998. [24] Middelburg, C. Logic and Specification: Extending Vdm-Sl for Advanced Formal Specification (Computer Science: Research and Practice). Chapman & Hall, 1993. [25] McMorran, M. A., Powell, S. Z Guide for Beginners. Blackwell Scientific, 1993. [26] Nissanke, N., Robinson, N. Formal Methods in Safety Analysis. In: V. Maggioli, editor, SAFECOMP’94, The 13th International Conference on Computer Safety, Reliability and Security, Anaheim, California, 1994. Instrument Society of America, pp. 239–248. [27] Nissanke, N., Dammag, H. Risk Bands – A Novel Feature of Safecharts. Proceedings of the 11th International Symposium on Software Reliability Engineering, ISSRE 2000, October 8–11, 2000, San Jose, California, USA, pp. 293–301. [28] Parnas, D. L., Asmis, G. J. K., Madey, J. Assessment of Safety-Critical Software in Nuclear Power Plants. Nuclear Safety, Vol. 32, No. 2, April – June 1991, pp. 189– 198. [29] RTCA/DO-178B. Software Considerations in Airborne Systems and Equipment Certification, RTCA, Washington DC, 1992. [30] Ruddle, A. Formal Methods in the Specification of Real-Time, Safety-Critical Control Systems. In: J. P. Bowen and J. E. Nicholls (Eds.), Z User Workshop, London, 1992. Proceedings of the Seventh Annual Z User Meeting, London, 14–15 December 1992, Springer-Verlag, 1993, pp. 131–146.

11

[31] Spivey, J. M. The Z Notation: A Reference Manual. Prentice Hall International Series in Computer Science, 2nd edition, 1992. [32] Stepney, S., Lord, S. Formal Specification of an Access Control System. Software – Practice and Experience, Vol. 17(9), September 1987, pp. 575–593. [33] Stepney, S. New Horizons in Formal Methods.The Computer Bulletin, BCS, January 2001, pp. 24–26. [34] TACS/1019/N7. User Guide for MALPAS Release 6.0. TA Consultancy Services Limited. The Barbican, East Street, Farnham, Surrey GU9 7TB: December 1992. [35] Valentine, S. Putting Numbers into the Mathematical Toolkit. In: J. P. Bowen and J. E. Nicholls (Eds.), Z User Workshop, London, 1992. Proceedings of the Seventh Annual Z User Meeting, London, 14–15 December 1992, Springer-Verlag, 1993, pp. 9–36. [36] Vilkomir S. A., Kharchenko V. S., Ponomaryev A. S., Gorda A. L. The System Safety Assessment by the Use of Programming Tools during the Licensing Process. – Proceedings of the 17th International System Safety Conference, Orlando, Florida, USA, August 1999, pp. 222–227. [37] Vilkomir, S. A., Kharchenko, V. S. Methodology of the Review of Software for Safety Important Systems. Safety and Reliability. Proceedings of ESREL’99 – The Tenth European Conference on Safety and Reliability, Munich-Garching, Germany, 13–17 September 1999, Vol. 1, pp. 593–596. [38] Vilkomir, S. A., Kharchenko, V. S. An ‘Asymmetric’ Approach to the Assessment of Safety-Critical Software During Certification and Licensing. Project Control: the Human Factor, Proceedings of ESCOM–SCOPE 2000 Conference, 18–20 April 2000, Munich, Germany, pp. 467–475. [39] Vilkomir, S. A., Bowen, J. P. Formalization of Control-flow Criteria of Software Testing, Technical Report SBU-CISM-01-01, SCISM, South Bank University, London, UK, January 2001. [40] Woodman, M., Heal, B. Introduction to VDM. McGraw-Hill, 1993. [41] Wordsworth, J. Software Engineering with the B-method. Addison-Wesley, 1996.

12