Page 1 of 8

Reliability Engineering Applied to Critical Operations Power Systems (COPS) Michael Anthony, PE University of Michigan ([email protected])

HP Critical Facility Services Robert Arno ([email protected]), Patrick Saad Saba ([email protected]) Robert Schuerger PE [email protected])

Abstract: At the request of the US Homeland Security Department in 2005 the National Fire Protection Association (NFPA) developed the first leading practice criterion for building premises wiring in emergency management facilities. These criteria first appeared in the 2008 National Electrical Code (NEC) as a new section -- Article 708: Critical Operations Power Systems (COPS). Article 708 establishes minimum design, commissioning and maintenance requirements for facilities with engineering documentation that identifies them as designated critical operations areas (DCOAs). One of the key features of Article 708 is the application of quantitative methods for assessing risk and conveying the results into a power system design that is scaled according to hazards present in any given emergency management district. These methods employ classical lumped parameter modeling of power chain architectures and can be applied to any type of critical facility, whether it is a stand-alone structure, or a portion of stand-alone structure, such as a police station or government center. This article will provide a risk assessment roadmap for a typical COPS facility that is most common -- a “911” Call Center (the facility that takes and routes the 911 calls to the police or fire departments). The existing methods of reliability engineering will be used in the risk assessment.

DLB Associates Mark Beirne ([email protected])

state, federal, or other codes by any governmental agency having jurisdiction or by facility engineering documentation establishing the necessity for such a system. The genesis of COPS was the widely perceived need to harden emergency and standby power systems that support homeland security operations. The 9/11 terrorist attacks and Hurricane Katrina revealed the need to reassess national electrical infrastructure protection and reliability at the building premises level where local practices vary widely. NFPA 1600, Standard on Disaster/Emergency Management and Business Continuity Programs contains parallel and complementary requirements that were also updated to better protect critical infrastructure. [2] Underwriters Laboratory Standard 827, Standard for Central Station and Fire Alarm Systems, also contains criterion for the security of privatelyowned data centers but because its criterion is more closely correlated with product-oriented underwriting for insurance companies -- rather than criterion intended for adoption by organizations involved in the rescue, response and recovery operations -- it will not be covered in this paper. [3]

Index Terms: reliability, availability, mean time between failure, FMECA, Fault Tree Analysis, Critical Operations Power Systems, Designated Critical Operations Area, emergency management, homeland security

Since a “911” Call Center is the critical communications link between the public and both the police and fire departments, the authors of this paper have assumed that most, if not all local emergency management agencies would require that 911 call centers be engineered, built, commissioned and maintained according to Article 708 requirements.

I. INTRODUCTION

II. NEC ARTICLE 708 OVERVIEW

In October of 2005 the NFPA Standards Council issued a directive to the Technical Correlating Committee of the NEC to prepare Article 708 for the 2008 NEC to define a new class of power system. The scope of Article 708 was permitted to extend beyond the NEC’s traditional scope that limited it to the “practical safeguarding of persons and property from hazards arising from the use of electricity” further into design, operation and maintenance criterion. [1] The NEC defines COPS as: Power systems for facilities or parts of facilities that require continuous operation for the reasons of public safety, emergency management, national security, or business continuity. The article also defines Designated Critical Operations Areas (DCOA) - Areas within a facility or site designated as requiring critical operations power. According to the NEC, COPS are designated by the “Authority Having Jurisdiction” (AHJ) typically municipal,

Article 708, Section I - General contains the following elements: 708.1 Scope 708.2 Definitions 708.3 Application of Other Articles 708.4 Risk Assessment 708.5 Physical Security 708.6 Testing and Maintenance 708.8 Commissioning Other Sections in Article 708 are as follows: II. III. IV.

Circuit Wiring and Equipment Power Sources and Connection Overcurrent Protection

978-1-61284-127-4/11/$26.00 © 2011 IEEE

Page 2 of 8

V.

System Performance and Analysis

This paper will focus primarily on 708.4 Risk Assessment, to provide typical techniques used in Reliability Engineering and show how it can be used to evaluate the reliability, and to scale the availability, of the COPS. Some of the other sections, such as Overcurrent Protection, also have an impact on reliability. Due to the space constraints, the analysis in this paper assumes proper selective coordination of all overcurrent protective devices, which is an important factor.

(4)

III. RISK ASSESSMENT A key concept of COPS engineering appears in Section 708.4:

(5)

Risk Assessment for COPS shall be documented and shall be conducted in accordance with 708.4 (A) thru (C) (A) Conducting Risk Assessment – Identify hazards, the likelihood of their occurrence and vulnerability of the electrical system to those hazards. (B) Identification of Hazards – Minimum shall include, but shall not be limited to, the following: (1) Naturally occurring hazards (geological, meteorological and biological) (2) Human-caused events (accidental and intentional) (C) Developing Mitigation Strategy – Based on the results of the risk assessment, a strategy shall be developed and implemented to mitigate hazards. There are a number of methodologies and techniques available for risk assessment that range from simple to complex. [4][5] The major types of these techniques are listed below. Items (1) thru (5) provide qualitative results; item (6) can be used to provide both qualitative and/or quantitative results; and item (7) provides primarily quantitative results. “What-if”: The purpose of the what-if analysis is to identify specific hazards or hazardous situations that could result in undesirable consequences. This technique has limited structure but relies on knowledgeable individuals who are familiar with the areas/operations/processes. The value of the end result is dependent on the team and how thorough they are in asking questions regarding potential hazards. (2) Checklist: A specific list of items is used to identify hazards and hazardous situations by comparing the current or projected situations with accepted standards. The value of the end result is dependent on the quality of the checklist and the skill and understanding of the checklist user. (3) What-if/checklist: This technique is a combination of the what-if and checklist techniques, and uses the strength of both techniques to complete the risk assessment. The what-if questions are developed and the checklist(s) used to encourage the creativity of the

(6)

(1)

(7)

what-if process, as well as fill in any gaps in the process of developing questions. The value of the end result is dependent on the team and how thorough they are in asking questions regarding potential hazards. Hazard and operability study (HAZOP): This technique requires an interdisciplinary team that is very knowledgeable of the areas/operations/processes to be assessed. This approach is thorough, timeconsuming, and costly. The value of the end result depends on the skill and understanding of the team, the quality of the reference material available, the ability of the team to function as a team, and strong, positive leadership. Failure Modes, Effects and Criticality Analysis (FMECA): Each element in a system is examined individually and collectively to determine the effect when one or more elements fail. This is a bottom-up approach: each of the elements is examined, all of the ways it can fail are listed (failure modes) and the effect of each failure to the element itself and on the overall system is predicted. Then a criticality level is assigned for each failure mode, base on the overall effect on the system. An interdisciplinary team is required and it is time consuming in direct proportion to how thorough and to what level of detail the analysis is taken. This technique is well suited for assessing potential equipment failures and how they impact the overall mission of the system being analyzed. The value of the end result is dependent on the skill and understanding of the team and scope of the analysis performed. Fault Tree Analysis (FTA): This is a top-down approach where an undesirable event is identified as the “top event” in the “tree” and the potential causes that could lead to the undesirable event are identified as “branches” below. Boolean Algebra is used to connect the potential causes of failure in the branches to other branches and the top event. If the failure rate and repair data is available for all of the initiating failures in the Fault Tree, quantitative results (unreliability and unavailability) can be calculated for the “top event” and each of the branches. The value of the assessment is dependent on the competence of the team in using the FTA process, on their skill and understanding of the systems they are analyzing, and on the depth to which they take the analysis. Reliability Block Diagram (RBD): An RBD is a block diagram in which the major components are connected together in the same manner as they are in a lumped parameter one-line or piping diagram. Each of the blocks have the failure and repair data for that component included in the block. The junctions connecting the block are set according to the system redundancy (e.g. “one out of two” when there are two components and only one is required to carry the load). Quantitative results (reliability, availability, etc.) for the RBD are obtained by performing the series and parallel combinations of the blocks.

978-1-61284-127-4/11/$26.00 © 2011 IEEE

Page 3 of 8

The risk assessment for COPS is performed to identify hazards, the likelihood of their occurrence, and the vulnerability of the COPS to those hazards. These hazards span a full range of hazards that include equipment failure, inclement weather, flooding, earthquakes, and civil disturbances. [Refer to NFPA 1600 Appendix 5.3 for a more comprehensive list of potential vulnerability parameters.] This could be called a vulnerability analysis. From the vulnerability analysis, the power distribution architecture, and supporting mechanical systems, can then be properly selected to achieve the desired reliability. [6]

Annex F

It is noteworthy that the hazards listed in 708.4 (B) are not just items that could cause the electrical system to fail. Obviously electrical systems are immune to biological hazards. NFPA 1600-2007, Standard on Disaster/Emergency Management and Business Continuity Programs, can be used as a model for Article 708 and is referenced in a number of places. Therefore the Risk Assessment should be comprehensive and look at all of the major areas that would take the “911” Call Center out of service.

 Availability – the percentage of time that a system is available to perform its function  Reliability – the probability that an item can perform its intended function for a specified interval under stated conditions  Maintainability – a measure of how quickly and economically failures can be prevented through preventive maintenance, or system operation can be restored following failure

For our example “911” Call Center, we are going to use a very robust design in which there are two complete systems for each part of the critical infrastructure. In the data center world this is referred to as “2N,” in which “N” is for “number” (needed). The critical systems are as follows:

Availability is calculated by

1. 2. 3. 4. 5. 6. 7.

Electrical power (ac) Electrical power (dc) Mechanical cooling system Telephone system Shortwave Radio System IT Systems including connection to Internet Building Life Safety Systems (structural, & fire protection)

Figure 1 shows the one-line diagram for our example facility. The IT equipment, shortwave radio and mechanical cooling systems are on ac power. The telephone system uses dc power. Figure 2 shows the mechanical cooling system. It is also “2N,” with one air-cooled chiller and one water cooled chiller. The water cooled chiller is more economical to operate, but requires water to make up for evaporation from the cooling tower. The air-cooled chiller can operate without make-up water, should the city water supply be lost. It should be noted that control systems play a major role in determining the Reliability of critical facilities and should be the third component to electrical and mechanical. The complexity of the control systems varies based on the size and complexity of the facility. For this reason and for a better understanding of this application control systems were not included.

Since many of the concepts underlying Article 708 cannot be crafted in mandatory, legislative language, an Annex F was included in the 2008 NEC to familiarize the premises wiring safety community -- the bulk of the users of the NEC - with some of the vocabulary of reliability engineering. Annex F also highlights the importance of proper installation, and commissioning of COPS systems. Availability and Reliability as defined in Annex F:

A

MTBF MTBF MTTR

 Mean time between failures (MTBF) – the average time the equipment performed its intended function between failures.  Mean time to repair (MTTR) – the average time it takes to repair the failure and get the equipment back into service. IV. RELIABILITY ANALYSIS IEEE Gold Book, Standard 493-2007, Recommended Practice for Design of Reliable Industrial and Commercial Power Systems [7] provides the methodology, along with failure and repair data required to perform reliability analysis on the electrical and mechanical systems. (Space does not permit listing actual data used). The method used in the IEEE Gold Book is Reliability Block Diagrams (RBD). RBD is an effective method to analyze systems with many items that are interrelated, such as an electrical distribution system. For simple systems that consist of series and parallel blocks, the calculations can be performed manually. For complex systems with standby components, such as in Figure 1, reliability software is needed. Shown in Table 1 is the reliability analysis for Figures 1 and 2 using reliability software to calculate the reliability, availability, MTBF and MTTR for each. There is a significant difference between what is considered a “failure” for electrical power to the critical ac loads and dc telephone systems when compared to “failure”

978-1-61284-127-4/11/$26.00 © 2011 IEEE

Page 4 of 8

Figure 1: One-line diagram of the 911 Call Center.

978-1-61284-127-4/11/$26.00 © 2011 IEEE

Page 5 of 8

for power to the mechanical cooling systems. Momentary loss of power to the critical ac loads and telephone systems is considered a “failure,” since UPS system and the 48 V dc systems have batteries to provide power while the generators start (on loss of utility power). For the electrical power to the mechanical cooling systems, momentary loss of power is not considered a failure, since the mechanical cooling system is designed to go down and restart. Description of RBD Electrical Power for Critical AC Loads Electrical Power for Critical DC Loads Electrical Power for Mechanical System Mechanical Cooling System

Failure Rate

MTTR (hours)

Availability

Reliability (5 years)

2.1464 E-06

6.31

0.9999865

7.70%

2.4037 E-06

5.92

0.9999858

9.03%

5.4260 E-07

2.19

0.9999988

4.30%

2.7977 E-07

The reliability analysis shown above for the mechanical cooling system just shows the reliability of the equipment itself. It does not address the probability that any equipment actually overheats. The COPS must operate for long periods of time, providing power to systems that perform critical functions. Section 708.22 (C) specifically requires that the alternate power source be capable of operating 72 hours at full load. Therefore the 911 Call Center would require significant onsite diesel fuel storage to be included in the design. V. ADDITIONAL TOOLS FOR THE RISK ANALYST

7.15

0.9999980

1.46%

Table 1: Reliability analysis for critical electrical and mechanical systems (Failure Rate = failures per hour of operation) There is also a significant difference between loss of power to the critical ac & dc loads and loss of cooling. Loss of power to the critical ac load or telephone system causes an immediate failure. Loss of cooling does not. The equipment has to overheat before it shuts down, which may take a significant amount of time.

Annex F of NEC 2008 also provides direction on how to improve the availability of COPS, both for existing facilities and new facilities. The methodology includes the reliability analysis of evaluating possible failures of the system by conducting failure modes and effects criticality analysis (FMECA) and/or a fault tree analysis (FTA). Because of the complexity of the modeling and interpretation of the risk analysis, Annex F references supporting documents. Army Corp of Engineers Power Reliability Enhancement Program (PREP) Training Manuals TM 5-698-4, Failure Modes and Effects Criticality Analysis (FMECA) for Command, Control, Communications, Computer, Intelligence, Surveillance, and Reconnaissance (C4ISR) Facilities 29 September 2006, provides the necessary information on the how to conduct a FMECA [8]. As the name states, a FMECA is the process of looking at all of the failure modes for the equipment or system being analyzed and determining what effect each failure would cause. There can be more than one effect and it is normal to list them as “primary” and “secondary,” etc. As a simple example, a molded case circuit breaker has the following major failure modes: 1. 2. 3.

Fail to open when it should Fail to close when it should Fail to conduct or stay closed when it should

The “when it should” part of each of the above creates a further breakdown of each. For example for the first (fail to open) can be for a number of reasons such as: 1. 2. 3. 4. Figure 2: Cooling system for “911” Call Center

Thermal unit is defective and did not detect the overload. The instantaneous (magnetic) element is defective and did not detect the short circuit. The mechanism is defective and pulling the handle did not separate the contacts. The contacts are welded shut from a previous failure and cannot be opened even manually.

978-1-61284-127-4/11/$26.00 © 2011 IEEE

Page 6 of 8

The FMECA process is continued until all of the major failure modes for all of the equipment (or systems) being evaluated have been listed out. Then the primary, secondary, etc. effects are listed out for each failure of each piece of equipment. In our above example, assume the circuit breaker was supplying a motor that pumped chilled water to Computer Room Air Handling (CRAH) units. The primary effect of the circuit breaker not tripping could be the motor keeps running and starts a fire (if it has a short circuit and the breaker up stream does not clear the fault) or it may mean the whole cooling system goes down when the main breaker trips to clear the fault. The secondary effect may be loss of some of the IT equipment being cooled by the CRAH units, which in turn may create further effects. If the cooling system has been designed with redundancy there may be no secondary effects, since loss of one pump may not cause any IT equipment to overheat. If a fire starts it may force the whole raised floor IT load to go down regardless of the cooling system. Once all the failure modes and effects have been defined, the next piece is to look at the criticality of the effects in conjunction with the probability of each one happening. The normal method is to have a gradient scale of criticality such as: I. Catastrophic (major human injury, significant financial loss, significant PR impact) II. Critical (significant loss of production, minor human injury) III. Marginal (minor loss of production) IV. Minor (no impact to production that is significant) The probability of each failure is then addressed. Another gradient scale is often used, such as: A. B. C. D. E.

Frequent Reasonably probable Occasional occurrence Remote Extremely unlikely

Then each failure and its associated effect is evaluated in terms of the above two scales of criticality and likelihood of occurrence and put into a matrix as shown in Figure 3. If your system has been very well designed, there will be no catastrophic events that are likely to occur. But that is often the point of doing the FMECA in the first place; to evaluate how well the system has been designed. The FMECA matrix points out which areas need to be addressed that will give the greatest impact to the overall operation of the system. It also shows which areas not to bother with, as

they are either extremely unlikely to occur or the effect when they do occur is minor. FMEA Criticality Matrix 10 8

Quantity

IEEE Standard 3007.2-2010, Recommended Practice for the Maintenance of Industrial and Commercial Power Systems also has a detailed example of a FMEA for a 480 V switchboard that demonstrates this process. [9]

I. Catastrophic 6

II. Critical III. Marginal IV. Minor

4 2 0 A- Frequent

CD - Remote BReasonably Occasional Probable Occurrence

EExtremely Unlikely

Figure 3: FMEA Criticality Matrix To perform a risk assessment using Fault Tree Analysis (FTA), the event that is to be investigated is placed at the top of the “tree.” Below the top event are the items that can cause the top event to occur. The Fault Tree uses Boolean Algebra, with OR gates (in which either event can cause a failure), AND gates (which require both events for the failure) and Initiating Events (which are events like equipment failures, to be evaluated).

Figure 4: Fault Tree Symbols: OR, AND, & Initiating Event Figure 5 presents the top part of a Fault Tree for COPS that can be used to perform a risk assessment. The top event is “COPS out of Service.” The hazards listed in section 708.4 (Naturally Occurring Hazards and Human Caused Events) have been diagramed, with the addition of the “equipment failure” as an additional source of the COPS out of Service. Equipment failure could have been included under “human caused events.” However, it is an item that can be specifically addressed with Reliability Analysis much more easily than other types of human caused events, such as operator error. Below each of the major hazards additional parts of the Fault Tree would be included. Shown in Figure 6 is the expansion of the Communications Equipment Failure. If the telephone system fails, communication with the outside world is lost. (CO = Central Office for telephone company) If the shortwave radio system is lost, communication with the police and fire fighters in the field is lost. Some 911 Call Centers may also consider loss of Internet communication a critical failure, but for this example we assumed the center could continue to function without it. This diagram shows all of the Initiating Events that are to be evaluated in the fault tree. The rate at which past failures have occurred (failure rate, FR) for each of the Initialing

978-1-61284-127-4/11/$26.00 © 2011 IEEE

Page 7 of 8

Figure 5: Fault Tree for COPS

Figure 6: Fault Tree showing the Initiating Events to be evaluated for Communication Equipment Failure Events would be listed in that figure. For example, Table 1 provides the failure rates for loss of ac or dc power to the shortwave radio or telephone system respectively. Once all of the failure rates have been determined for all of the initiating events, the fault tree can be calculated and the reliability and availability of the Fault Tree determined. Obtaining a failure rate for some of the items would be relatively easy, since the IEEE Gold Book provides data on a wide range of electrical and mechanical equipment. However, for some of the items, such as Infectious Agent under Biological, it would be much more difficult to determine a failure rate. Human caused failures, whether intentional (sabotage) or accidental is another area that is very difficult to quantity. In areas where failure data is not available, direct reliability and availability cannot be performed. Mitigation strategies, such security systems and preventing access to the COPS or its support systems, will have to address these types of issues without the assistance of reliability analysis.

Reliability analysis can be done by several different methods. Fault Tree Analysis is quite effective in analyzing a system in which a number of factors (that are relatively independent of each other) can cause a system failure. It is also very useful in showing the relationships between the systems. However, it should be noted that even with the best engineering design and technology it may be economically impractical and technically impossible to design, build and maintain a COPS that will never fail over a long period of operation. Forced outages may and do occur. That is why the original authors of Article 708 included COPS maintenance and commissioning requirements of Sections 708.6 and 708.8 in the enforceable text of the NEC. When forced outages do occur, restoring the COPS to operation as quickly and economically as possible is very important. Thus the maintainability characteristics of the COPS will predict how quickly and economically it will be restored to normal operation. That is why reliability, availability and maintainability (RAM) are considered complementary characteristics. [10][11] VI. CONCLUSION

978-1-61284-127-4/11/$26.00 © 2011 IEEE

Page 8 of 8

Article 708 leaves much to the judgment of the engineer designing the COPS. The quantitative approaches described in this paper will lead to more consistency in those judgments by conveying opinions about power security into the realm of science. The risk assessment method chosen to analyze a COPS should be correlated with the hazards in any given emergency management district and be appropriate for the system in question. It should require only a reasonable level of investment given the value of the results. The failure of some components may have little impact on either system function, or on its operating repair costs. Given the ranking of hazards in any given emergency management district, and the subtlety in the performance of the power chain architecture, a relatively costly analysis may not be justified. In any case, when the consequences of failure are catastrophic, every possible effort should be made to make the COPS fail-safe. What would be considered “an appropriate risk assessment” would be one that economically directed the local emergency management agencies on how to best spend the limited funds they have on what the most effective upgrades to the facilities would be. Article 708 was added to the NEC "Special Conditions" chapter when the NFPA realized the need and the means to convey the best practices of the business continuity industry into the public sector emergency preparedness. In general, building codes (NFPA 101, NFPA 5000, IBC, etc) tend to mandate requirements that contribute to public safety and -to a lesser extent -- property protection. Article 708, and its optional supporting material presents a performance-based design approach that is more suited for the spectrum of COPS facilities, and their diverse users. There are still significant jurisdictional issues to be worked out before Article 708 conformity becomes the main driver for increased homeland security funding at the building premises level.[12] The wisdom in placing COPS requirements into the NEC is that its presence would instantly be discussed in building departments among local authorities having jurisdiction and emergency management functionaries. It has requirements that are clearly outside the scope of the normal practice of building premises wiring. Other documents, such as NFPA 1600 -Standard on Disaster/Emergency Management and Business Continuity Programs and NFPA 110 - Standard for Emergency and Standby Power Systems will have to be revised to align and support the intent of COPS. Looking forward to future revision cycles of the NEC, a logical next step would be to define scalable levels of COPS to match the different levels of criticality of the various types of facilities. For example, a facility required to provide emergency communication across a large area, such as a 911 Call Center, would be more important to public safety than an individual police or fire station. Therefore the 911 Call

Center should have more robust COPS than what would be necessary for the individual police or fire station. The AHJ should also be provided with some guidelines for assessing acceptable reliability and availability for the various types of facilities critical to homeland security. It should be noted that the analysis presented in this paper is based on the 911 call center being the only source of support for the region it operates in. If there are multiple 911 call centers covering the same region then the risk assessment should include the effects of back up centers – or redundant centers providing duplicate service. REFERENCES 1. “When Failure Isn’t an Option” Jim Lardear, NEC Digest, February 2007 2. NFPA 1600, Standard on Disaster/Emergency Management and Business Continuity Programs”, National Fire Protection Association, Quincy, MA 3. UL 827 – Standard for Safety Central Station Alarm Services – Underwriter’s Laboratories, Northbrook, IL. 4. “NEC 708” Michael A. Anthony, Consulting-Specifying Engineer, May 2007. 5. “Article 708: Critical Operations Power Systems” Electrical Construction & Maintenance Magazine, November 2007, by Michael A. Anthony, Robert G. Arno, Evangelos Stoyas. 6. “Risk Assessments for Critical Operations Power Systems” Michael A. Anthony, Robert Arno, Robert Schuerger and Evangelos Stoyas Pure Power Magazine, June 2008 7. IEEE Standard 493 – 2007, Recommended Practice for Design of Reliable Industrial and Commercial Power Systems. 8. Army Corp of Engineers Power Reliability Enhancement Program (PREP) Training Manuals TM 5-698-4, Failure Modes and Effects Criticality Analysis (FMECA) for Command, Control, Communications, Computer, Intelligence, Surveillance, and Reconnaissance (C4ISR) Facilities 29 September 2006 9. IEEE Standard 3007.2 – 2010, Recommended Practice for the Maintenance of Industrial and Commercial Power Systems. 10. “Article 708, Critical Operations Power Systems – Some Existing Technologies to Assist in Complying,” by Robert Arno, Robert Schuerger and Evangelos Stoyas, IAEI Magazine Nov.-Dec. 2008 11. “Risk analysis for NEC article 708 Critical Operations Power Systems,” by Robert Arno, Robert Schuerger and Evangelos Stoyas, IEEE 2009 IAS Conference Proceedings 12. “Critical Operations Power Systems: Success of the Imagination”, Michael A. Anthony PE & Richard Aaron JD. International City-County Management Magazine, January/February 2009

978-1-61284-127-4/11/$26.00 © 2011 IEEE