Establishing Formal Regulatory Requirements for Safety-Critical Software Certification Sergiy A. Vilkomir and Jonathan P. Bowen South Bank University Centre for Applied Formal Methods School of Computing, Information Systems and Mathematics 103 Borough Road, London SE1 0AA, UK Email: {vilkoms, bowenjp}@sbu.ac.uk URL: http://www.cafm.sbu.ac.uk/ Tel: +44-(0)20-7815-7079 Fax: +44-(0)20-7815-7499 Abstract Formal methods are usually used for computer system specification, production and verification. In this paper, a new direction for the use of formal methods is proposed, namely formalization of the regulatory requirements for software of safety-critical control systems. Formal regulatory requirements, as opposed to formal specifications for a concrete system, have a generic nature, are applicable to a wide range of safety-critical control systems and could act as the basis for the certification or licensing process. The use of formal regulatory requirements could help to eliminate ambiguity or misunderstanding of informal definitions, to allow rigorous assessment of satisfaction with respect to requirements and finally to increase the safety level of a system. As a formal notation for describing the regulatory requirements, the Z notation is proposed. To demonstrate the approach, requirements for protection against common mode software failures and requirements for protection against unauthorized access are considered. Z schemas for these requirements are presented. Keywords: certification, formal methods, regulatory requirements, safety-critical systems, software standards, Z notation 1 Introduction Formal methods make available a methodological framework for system specification, production and verification. These methods provide the means of development and assessment of a system or parts of a system (e.g., software), based on principles such as logical reasoning, set theory, abstract algebra or graph theory, that allow mathematical manipulation and proof of properties. Formal methods are especially recommended for safety-critical applications [6], where high integrity is important. The designer of control system and software has a major responsibility for ensuring safety. The application of formal methods to safety-critical systems from the point of view of designers, i.e., for system or software development, is considered in many scientific and technical publications (e.g., see [5, 15]).

In many countries, in areas such as air transport and nuclear energetics, there are also special state regulatory bodies (authorities) whose basic tasks are independent control of safety, certification and licensing of activity connected with safety. Concerning computer control systems, there are two main tasks of the regulatory body: -

The establishment of regulatory requirements for systems and software; i.e., requirements (criteria, norms, rules) important for safety; An assessment of the system and software during certification or licensing (regulatory assessment). The purpose of this assessment is to be convinced that the system and software answer established regulatory requirements.

These tasks are complex, labour consuming, and have a specific character. It is expedient to use the special methods and tools for the decisions concerning these tasks. The application of formal methods during certification has been considered to a much lesser degree than for software development and is not widely put into practice at present. Nevertheless, the application of formal methods in this area has potential to be useful and successful. When formal methods have been used during certification and licensing, it was mainly for systems or software assessment, i.e., for solving the second of the tasks above. Thus, with regard to using formal methods, application of software tools for static and dynamic analysis could be considered. Examples of using such tools for regulatory assessment during the certification or licensing process include using MALPAS [34] for the French nuclear plants protection system, SPIN software assessment [14], and LDRA Testbed and VALIDATOR tools for the independent assessment of the Teleperm XS software [23]. A major example of a manual formal approach was using “function tables” for safety assessment of software for the shutdown systems of the Darlington Nuclear Plant in Canada by Parnas and others [28]. Various approaches to using formal methods in safety analysis, which can be useful for the regulatory assessment of safety-critical systems, are considered in [2, 9, 12, 16, 26, 30]. In this paper, a new direction is proposed, namely the formalization of the regulatory requirements for software in control systems. This involves the use of formal methods for establishing these regulatory requirements (the first of the mentioned tasks above). Formal regulatory requirements, as opposed to formal specifications for a particular implemented system, have a generic nature and could be applicable to a wide range of safety-critical control systems and software. The use of formal regulatory requirements could help to eliminate various difficulties in the understanding of problems or ambiguity of informal definitions, allowing rigorous assessment of satisfactory implementation with respect to requirements and finally to increase the safety level of a system. This approach is described in the next section. 2 Formalization of Regulatory Requirements for Software 2.1 General approach The regulatory requirements can be determined in: -

normative documents of the regulatory authority; national standards; international standards (IEC, ISO, etc.); recommendations of international organizations (IAEA, etc.).

It is possible to divide the regulatory requirements for control system software into various areas, for example [37] requirements for: -

the structure and elements of software; the diagnostics; protection against failures and distortions; software development procedures; software testing.

The regulatory requirements are usually the general requirements for software in safetycritical systems, irrespective of their functionality (e.g., for all safety-critical control systems at nuclear power plants). These systems have a range of purposes, have differing levels of completeness and sizes, and are developed by different designers using various methods and techniques. Thus, the requirements can possess contradictory characteristics. On the one hand, they should be general enough to leave a designer a free choice of techniques; i.e., they should not be dependent on methods of development. On the other hand, the requirements should be specific enough to show a designer what should be done and provide guidance on how it should be done to satisfy the requirements. It is not always easy to formulate the requirements in such a manner. For example, informal requirements (expressed in natural language) have the following shortcomings: -

They are too general and do not contain instructions on how it is possible to satisfy and to check these requirements; They are not accurate enough, allowing various possible interpretations requiring additional explanation.

To alleviate these problems, we propose the use of formalized regulatory requirements. The formalized requirements (i.e., elaborated using a formal mathematically-based notation) can help eliminate the above-mentioned shortcomings. The formal definitions are more specific and precise, and help ensure common understanding of the requirement contents both between experts within a regulatory authority and between experts and designers. The structure of a formal definition can also facilitate checking the realization of the requirements. Another benefit of using formal requirements is the possibility of formalizing and automating the regulatory assessment process. The use of formal requirements facilitates a special software tool application both for support of the whole licensing process [36] and for an individual requirement assessment. Sometimes problems with informal requirements arise when the system’s designer from one country develops a safety-critical control system for use in another country. The system should satisfy the regulatory requirements of the country where this system will be in operation. However, the designer may be not familiar with these requirements in detail. There could be different interpretations of the same requirements in various counties, even different meanings of the same terms in various languages. In this case, the formal requirements can be used as an “international language” and facilitate mutual coordination between a regulatory authority and a foreign system’s designer. Obviously, the understanding of formal definitions of requirements can be more difficult than definitions in natural language. The use of formal definitions can require special training to attain a sufficient professional standard. Unfortunately, there is no one common and universally recognized formal method or notation, which is followed by the majority of software developers. That is why we propose to use formal requirements not instead of but in parallel with the

requirements in natural language. For example, within a normative document of software requirements, the formal definitions can form a recommended supplement and be used only for restricted cases involving the most important and complicated requirements. The formal definitions of requirements could be a good base for discussion between experts of a regulatory authority and a software designer prior to development beginning. They could help to reach common understanding with respect to completing the licensing process, the contents of the regulatory requirements for the particular system (software), and the associated documentation, which should be submitted for the regulatory review. The selection of an appropriate formal notation is a separate problem. Generally, many of the well-known formal methods (with associated languages/notations) can be used one way or another for formalization of regulatory requirements for software; e.g., formal methods such as VDM [24, 40] and B [1, 41] or semi-formal methods like UML [3, 11], Statecharts [13, 17] and its variety Safecharts [27] could be applicable. In this paper, we propose to formalize requirements in the Z notation [31]. The choice of the Z notation has been made for the following reasons: -

The main purpose of Z is for specifying and designing software so Z is suitable for describing both general and detailed requirements for software; There are many examples of substantial and successful practical applications of the Z notation to safety-critical areas [5, 15]; Many textbooks on Z are now available (e.g., [4, 20, 21, 25]); Z is one of the most widely taught formal methods [7]; The community of Z users is well organized; international Z user meetings are held on the regular basis (e.g., [8]); Z itself is undergoing international ISO standardization; this is at the Final Committee Draft stage and is thus almost complete.

Formal regulatory requirements can be applied not only directly to software but also to the process of software development. Thus, one of the most important stages of regulatory assessment is the assessment of a software testing process [38]. In this case, the testing criteria can be used as regulatory requirements. For example, according the DO-178B standard [29], the testing process must correspond to Modified Condition/Decision Coverage Criterion (MC/DC). Software should satisfy this standard in order to be certified by the FAA (Federal Aviation Administration, USA), so this requirement is regulatory. The formal definition of MC/DC in Z notation is given in our paper [39], where Z schemas for another primary control-flow testing criteria are also introduced. This formal definition is precise and eliminates inaccuracy, which is intrinsic in definitions using natural language. Two examples of using the Z notation for formalization of regulatory requirements, typical for control systems, are considered below: namely requirements for protection against unauthorized access (section 2.2) and also against common mode failures (section 2.3). 2.2 Requirements for protection against unauthorized access The requirements for protection against unauthorized access with reference to software have specific features for each type of computer system. For example, for bank systems and databases the most important aspect is the protection against deliberate attempts of unauthorized access. For real-time control systems, this task is usually achieved by administrative and organizational measures and only special personnel operate such systems. So, the task of control system software is protection mainly against unpremeditated unauthorized actions of personnel.

Broad ranges of specialists (programmers, technologists, system engineers, operators) take part in maintenance of a real-time control system. An unpremeditated fault during the changing of program code or operation and alarm settings can have grave consequences from a safety point of view. Therefore, it is very important to differentiate the responsibilities of personnel according administrative functions and associated duty regulations. Software protection against unauthorized access should allow every member of staff to carry out only a limited set of actions during maintenance of a system in conformity with their personnel duties. This approach is not directly connected with a time factor but it reflects the specific features of maintenance of control systems. It is expedient to reflect this part of a problem in the regulatory requirements for software in such systems. However, many normative documents and standards include this requirement only in a very general form. For example, in the IEEE Std 603-1998 Criteria for Safety Systems for Nuclear Power Generating Stations [18], the requirement for control of access is formulated in only two sentences: “The design shall permit the administrative control of access to safety system equipment. These administrative controls shall be supported by provisions within the safety systems, by provision in the generating station design, or by a combination thereof.” The IEEE Std 7-4.3.2-1993 Criteria for Digital Computers in Safety Systems of Nuclear Power Generating Stations [19], which specifies additional computer-specific requirements, indicates that no requirements beyond IEEE Std 603 are necessary for control of access. But such general requirements may not be constructive enough for beneficial use during regulatory assessment. On the other hand, detailed explanations in natural language are not always possible in standards and are not always readable and well defined. So, it is interesting to consider the possibility of using of formal regulatory requirements for protection against unauthorized access. Formal methods have been used for the specification of access control of various systems. For example, the Z notation was used in the development of formal models for an access control systems for the ADMIRAL project [32]. More recently, VDM and PVS were used in the analysis of the requirements for an access control system, which is part of a comprehensive security system [10]. For the Mondex electronic cash system and the Multos secure smartcard operating system, the Z notation was used to specify the high-level abstract formal specification of the desired security properties, to specify the lower level design, and to provide formal proof of correspondence between the two levels [33]. Formal specifications for the above-mentioned system are large in size, reflect specific features, and obviously could not be considered as regulatory requirements. Below we examine how the discussed examples in this subsection differentiate the responsibility of personnel and, accordingly, differentiate the possibility of access, and how a generalization of this can be formalized in the Z notation as a regulatory requirement. The following given sets are used to establish this requirement: [NAME, LEVEL, ACTION] The first two sets are used for describing names of personnel and levels of access. The set ACTION includes the various actions that are available for personnel during operation and maintenance of a system (e.g., information reading or scanning, operation or alarm setting changes, technological algorithm modernization, program code replacement, etc.). Details of the contents of ACTION should be considered for each specific system and is not a subject of the regulatory requirements. The schema below (all schemas in this paper were checked using the ZTC type-checker package [22]) determines the requirement, where the set staff records the operating and technical personnel, and the set levels describes different levels of permitted access. Differentiation of

allowed access is described by two functions. The stafflevel function determines the level of access for each member of personnel (it is possible, of course, that several members have the same level of access). The actions function determines the contents (available actions) for each level. Usually, the actions associated with the various levels are different; i.e., the actions function is an injection. __ access __________________________

| staff : P1 NAME | levels : P1 LEVEL | stafflevel : NAME —||—> LEVEL actions : LEVEL >—||—> P ACTION |______________ | | dom stafflevel = staff | ran stafflevel = levels dom actions = levels ||___________________________________ In spite of its simplicity, this schema is a constructive requirement; i.e., it also determines how this requirement can be checked and confirmed. Thus, to confirm realization of this requirement, the user of specific control systems should submit to the regulatory authority the following information: -

possible actions during operation and maintenance of the system; levels of access with the subset of actions allowed for each level; a list of names (positions) of staff with the level of access for each.

The given schema does not cover all aspects of the requirements for access control. For example, the requirements for the identification of personnel (passwords) should be assigned. Here we show the way in which the formal requirements, while maintaining the necessary commonality, can be constructive and easily checked during regulatory assessment. 2.3 Requirements for protection against common mode failures Protection against failures (fault-tolerance) is one important technique that can help ensure a high level of reliability of a control system. In order to guarantee tolerance against failures of hardware, multi-channel redundant control systems are typically used (for example, three-channel systems with “2 from 3” logical voting of signals). However, this approach does not guarantee required protection against software failures. In the case of a software error, identical programs in different channels can have a simultaneous failure. It is possible to ameliorate software failures with the help of software itself, so the following regulatory requirement is very important for safety-related control systems: software should carry out protection against common mode failures, including failures caused by software errors. For protection against common mode software failures, the following approaches are possible [37]: -

The use of two different computer systems (primary and diverse) for the fulfilment of the same functions, but constructed using different hardware and software. For example, primary and diverse protection systems are parts of NPP Temelin (Czech Republic) design modification project;

-

The use of various software implementations in different channels of the system for the fulfilment of the same functions; The use of more than one criterion for system actuation in each situation, when the action of the system is necessary (functional redundancy).

There are different opinions concerning the expediency of these approaches because each one has its own advantages and disadvantages. The use of the latter approach (functional redundancy) is possible for real-time systems, which work in a waiting mode, i.e., when the actuation of the system is required in special (emergency) situations. An example of such system is a nuclear reactor protection (shutdown) system. The use of functional redundancy means that each emergency situation (the initial event for the system actuation) should be determined in at least two ways based on various technological parameters. For example, if the first way is based on the measuring of temperature (the main criterion), the second way could be based on the measuring of pressure (an additional criterion). In this case, if the system actuation did not take place due to errors in the part of the software that realized the main criterion of an actuation, then the system should operate owing to realization of the additional criterion. Using this requirement for real-time system software in practical regulatory assessment would benefit from a more precise definition. Below we consider how this requirement can be formally described using the Z notation. The following given set contains the names of all possible input parameters of a system: [IDENTIFIER] It should be stressed that this set is not primordially settled like a set of all input parameters from a system specification. We consider this set to also contain names of all potential parameters; i.e., if it is necessary to add a new input parameter during system development, the name of this parameter is in the set IDENTIFIER. The following definitions are also used in Z schemas where R denotes the set of real numbers (real numbers definitions in Z are addressed in [35]): R1 = = { x : R | xD0 } Time = = R1 Command ::= wait | act I.e., time is modelled as non-negative real numbers and there are two commands. The following schema describes the notion of an input parameter of a system: __ parameter ______________________

| indent : IDENTIFIER | areaparam : P1 R | valueparam : Time —> R |______________ ran valueparam P areaparam ||__________________________________ Each input parameter contains the name ident from IDENTIFIER. For any point in time t, the parameter has the value valueparam(t). The subset of real numbers areaparam specifies a range of the input parameter; i.e., for all t, valueparam(t) is a member of areaparam (which must be nonempty).

The following schema determines the criterion for a system actuation: __ criterion ____________________________________________________ | seqparam : seq parameter | num : N1 >> Command | statecrit : Time —— | areaparam : P1 (seq R) |______________ | # seqparam = num | } point : areacrit ● # point = num | } point : areacrit; i : 1..num ● point(i) V (seqparam(i)).areaparam | } t0 : Time ● statcrit(t0) = act ⇔ { i : 1..num ● i |→ (seqparam(i)).valueparam(t0) } V areacrit ||______________________________________________________________ Each criterion is based on a set of input parameters (the sequence seqparam); the strictly positive natural number num gives the number of these parameters. At any point in time, the criterion has one of the two possible values from elements in the set Command. The value wait means that no actions are necessary; i.e., the situation is not emergent. The value act means the occurrence of an emergency event when the system actuation is required. The state of the criterion depends on a combination of input parameters values. The set areacrit, which is the subset of sequences of num sets of real numbers, determines the area of combinations of input parameters where the system actuation is required. For most of the time, the state of the criterion is wait. On the other hand, situations for system actuation must always exist; i.e., the function statecrit is generally surjective and both commands must be possible at different points in time. The schema below determines a potential regulatory requirement for protection against common mode software failures. The set setmaincrit contains the main criteria of system actuation, which determines the system operation. The functional redundancy is realized by additional criteria (the set setaddcrit). In this connection, the correspondence between the main criteria and the associated additional criteria should be determined (by the function addcrit). The main requirement for the functional redundancy realization consists of ensuring the independence between any main criteria and adequate additional criteria. This means that these criteria should be based on different input parameters; i.e., for any main criterion maincrit, the sets maincrit.seqparam and (addcrit(maincrit)).seqparam should be disjoint.

__ FunctionalRedundancy ______________________________________

| setmaincrit, setaddcrit : P1 criterion | delay : criterion —||—> Time | addcrit : criterion —||—> criterion |______________ | # setaddcrit C # setmaincrit | dom delay = dom addcrit = setmaincrit | ran addcrit = setaddcrit | } maincrit : criterion | maincrit V setmaincrit ● maincrit.seqparam ∩ (addcrit(maincrit)).seqparam = Ž | | } maincrit : criterion; t0 : Time | maincrit V setmaincrit ∧ | maincrit.statecrit(t0) = act ∧ | (} t : Time | t < t0 ● maincrit.statecrit(t) = wait) ● | Y t1 : Time | t1 D t0 ● | (addcrit(maincrit)). statecrit(t1) = act ∧ | t1 1 t0 C delay(maincrit) ||___________________________________________________________

For functional redundancy of real-time systems, the factor of time is very important. The problem is that the real-time system actuation according to the additional criterion could have a response lag after initiation of an emergency situation. When this response lag is admissible or not depends on the specific emergency situation and the specific real-time system. So, during the functional redundancy realization, the admissible time delay in the system actuation delay(maincrit) should be established for each main criterion. The value of this delay can vary from fractions of a second to sizeable time intervals. The requirement for the functional redundancy realization is that the gap between time t0 of the emergency situation appearance according the main criterion and time t1 of the system actuation according the additional criterion should be less than the previously determined delay(maincrit). The presented Z schemas could be used practically for the carrying out of regulatory assessment. In this connection, it is necessary to concentrate attention on validity of the admissible time delays for the main criteria and expected time delays for additional criteria. 3 Conclusion and Future Work The subject of this paper is a proposal for a new direction in the use of formal methods, namely formalization of the regulatory requirements for software used in safety-critical control systems. Formal regulatory requirements can be used as a basis for certification and licensing when used in parallel with the requirements in natural language. The Z notation [4, 31] is proposed for the formalization of regulatory requirements. Z schemas have been elaborated for some example regulatory requirements typically found in control systems, namely requirements for protection against unauthorized access and protection against common mode software failures. These strict formal definitions could be used practically for aiding regulatory assessment. The proposed schemas are constructive requirements (i.e., they determine how these requirements can be checked and confirmed). Future work in this area will be directed at selecting requirements for which formalization would be the most effective, producing a set of formal definitions of these requirements, and consideration of the practical application of the elaborated approach. Another direction for future work could be the development of assessment methods to checking conformance with formal requirements.

References [1] Abrial, J.-R. The B-Book. Cambridge University Press, 1996. [2] Atchison, B., Lindsay, P., Tombs, D. A Case Study in Software Safety Assurance Using Formal Methods. Technical Report No. 99-31, Software Verification Research Centre, School of Information Technology, The University of Queensland, Australia, September 1999. [3] Booch, G., Jacobson, I., Rumbaugh, J. The Unified Modeling Language User Guide. The Addison-Wesley Object Technology Series. Addison-Wesley, 1998. [4] Bowen, J. P. Formal Specification and Documentation Using Z: A Case Study Approach. International Thomson Computer Press, 1996. [5] Bowen, J. P., Hinchey M. G. The Use of Industrial-Strength Formal Methods. Proceedings of 21st International Computer Software and Application Conference (COMPSAC’97), Washington DC, USA, 13–15 August 1997, pp. 332–337. [6] Bowen, J. P. The Ethics of Safety-Critical Systems. Communications of the ACM, Vol. 43, No. 4, April 2000, pp. 91–97. [7] Bowen, J. P. Experience Teaching Z with Tool and Web Support. ACM SIGSOFT Software Engineering Notes, Vol. 26, No. 2, March 2001, pp. 69-75. [8] Bowen, J. P., Dunne, S., Galloway, A., King, S. (Eds.). ZB 2000: Formal Specification and Development in Z and B. First International Conference of B and Z Users, York, UK, 29 August – 2 September, 2000. Proceedings. Lecture Notes in Computer Science, Vol. 1878, Springer-Verlag, 2000 [9] De Lemos, R., Saeed, A., Anderson, T. Analyzing Safety Requirements for Process-Control Systems. IEEE Software, Vol. 12, No. 3, May 1995, pp. 42–53. [10] Droschl, G. Analyzing the Requirements of an Access Control using VDMTool and PVS (abstract). In J. M. Wing, J. Woodcock, and J. Davies, editors, FM’99: Formal Methods, World Congress on Formal Methods in the Development of Computing Systems, Toulouse, France, September 1999. Lecture Notes in Computer Science, Vol. 1709, page 1870, Springer, 1999. [11] Gomaa, H. Designing Concurrent, Distributed, and Real-Time Applications with UML. Addison-Wesley, 2000. [12] Hansen, K., Ravn, A., Stavridou, V. From Safety Analysis to Software Requirements. IEEE Transactions on Software Engineering, Vol. 24, No. 7, July 1998, pp. 573–584. [13] Harel, D., Politi, M. Modeling Reactive Systems with Statecharts: The Statemate Approach. McGraw-Hill, 1998. [14] Henry, J. Y., Soubies, B., Le Meur, M., Boulc’h, J. Evaluation Methods for the I and C Safety Softwares of Nuclear Power Plants. International Atomic Energy Agency Technical Committee Meeting on Safety Assessment of Computerized Control and Protection systems. Ref: J4-TC-698.2, 12–16 October 1992, IAEA Headquarters, Vienna, Austria. [15] Hinchey, M., G., Bowen, J. P., editors. Industrial-Strength Formal Methods in Practice. Springer-Verlag, FACIT series, 1999. [16] Horl, J., Aichernig, B. Validating Voice Communication Requirements Using Lightweight Formal Methods. IEEE Software, Vol. 17, No. 3, May/June 2000, pp. 21–27.

[17] Horrocks, I. Constructing the User Interface with Statecharts. Addison-Wesley, 1999. [18] IEEE Std 603-1998. IEEE Standard Criteria for Safety Systems for Nuclear Power Generating Stations. 1998. [19] IEEE Std 7-4.3.2-1993. IEEE Standard Criteria for Digital Computers in Safety Systems of Nuclear Power Generating Stations. 1993. [20] Imperato, M. An Introduction to Z. Chartwell-Bratt, 1991. [21] Jacky, J. The Way of Z: Practical Programming with Formal Methods. Cambridge University Press, 1997. [22] Jia, X. ZTC: A Type Checker for Z Notation. User’s Guide. Version 2.03, August 1998. Division of Software Engineering, School of Computer Science, Telecommunication, and Information Systems, DePaul University, USA, 1998. [23] Linder, A., Miedl, H. Methodology and Tools for Independent Verification and Validation of Computerized I & C Systems Important to Safety. In: IAEA-IWG-NPPCI-98/1 Working Material. Computerized Reactor Protection and Safety Related Systems in Nuclear Power Plants. Proceedings of a Specialists’ Meeting, 27–29 October 1997, Budapest, Hungary, pp. 127–138. Reproduced by the IAEA, Vienna, Austria, 1998. [24] Middelburg, C. Logic and Specification: Extending VDM-SL for Advanced Formal Specification (Computer Science: Research and Practice). Chapman & Hall, 1993. [25] McMorran, M. A., Powell, S. Z Guide for Beginners. Blackwell Scientific, 1993. [26] Nissanke, N., Robinson, N. Formal Methods in Safety Analysis. In: V. Maggioli, editor, SAFECOMP’94, The 13th International Conference on Computer Safety, Reliability and Security, Anaheim, California, 1994. Instrument Society of America, pp. 239–248. [27] Nissanke, N., Dammag, H. Risk Bands: A Novel Feature of Safecharts. Proceedings of the 11th International Symposium on Software Reliability Engineering, ISSRE 2000, 8–11 October 2000, San Jose, California, USA, pp. 293–301. [28] Parnas, D. L., Asmis, G. J. K., Madey, J. Assessment of Safety-Critical Software in Nuclear Power Plants. Nuclear Safety, Vol. 32, No. 2, April–June 1991, pp. 189–198. [29] RTCA/DO-178B. Software Considerations in Airborne Systems and Equipment Certification, RTCA, Washington DC, USA, 1992. [30] Ruddle, A. Formal Methods in the Specification of Real-Time, Safety-Critical Control Systems. In: J. P. Bowen and J. E. Nicholls (Eds.), Z User Workshop, London, 1992. Proceedings of the Seventh Annual Z User Meeting, London, UK, 14–15 December 1992, Springer-Verlag, 1993, pp. 131–146. [31] Spivey, J. M. The Z Notation: A Reference Manual. Prentice Hall International Series in Computer Science, 2nd edition, 1992. [32] Stepney, S., Lord, S. Formal Specification of an Access Control System. Software – Practice and Experience, Vol. 17, No. 9, September 1987, pp. 575–593. [33] Stepney, S. New Horizons in Formal Methods. The Computer Bulletin, BCS, January 2001, pp. 24–26. [34] TACS/1019/N7. User Guide for MALPAS Release 6.0. TA Consultancy Services Limited. The Barbican, East Street, Farnham, Surrey GU9 7TB, UK, December 1992.

[35] Valentine, S. Putting Numbers into the Mathematical Toolkit. In: J. P. Bowen and J. E. Nicholls (Eds.), Z User Workshop, London, 1992. Proceedings of the Seventh Annual Z User Meeting, London, 14–15 December 1992, Springer-Verlag, 1993, pp. 9–36. [36] Vilkomir S. A., Kharchenko V. S., Ponomaryev A. S., Gorda A. L. The System Safety Assessment by the Use of Programming Tools during the Licensing Process. Proceedings of the 17th International System Safety Conference, Orlando, Florida, USA, August 1999, pp. 222–227. [37] Vilkomir, S. A., Kharchenko, V. S. Methodology of the Review of Software for Safety Important Systems. Safety and Reliability. Proceedings of ESREL’99: The Tenth European Conference on Safety and Reliability, Munich-Garching, Germany, 13–17 September 1999, Vol. 1, pp. 593–596. [38] Vilkomir, S. A., Kharchenko, V. S. An “Asymmetric” Approach to the Assessment of SafetyCritical Software During Certification and Licensing. Project Control: the Human Factor, Proceedings of ESCOM–SCOPE 2000 Conference, 18–20 April 2000, Munich, Germany, pp. 467– 475. [39] Vilkomir, S. A., Bowen, J. P. Formalization of Software Testing Criteria using the Z Notation, Proceedings of COMPSAC 2001: 25th IEEE Annual International Computer Software and Applications Conference, Chicago, Illinois, USA, 8–12 October 2001. IEEE Computer Society Press, 2001, pp. 351–356. [40] Woodman, M., Heal, B. Introduction to VDM. McGraw-Hill, 1993. [41] Wordsworth, J. Software Engineering with the B-method. Addison-Wesley, 1996.