The FAA’s Integrated Capability Maturity Model draws from 10 major sources, including several CMMs and ISO 9001, which makes it easier to improve processes across disciplines. Linda Ibrahim and Arthur Pyster

A Single Model for Process Improvement Lessons Learned at the US Federal Aviation Administration

T

he IT systems that underlie the US Federal Aviation Administration’s operation are among the most complex in the world. On a typical day, the FAA safely guides the flight of nearly 2 million passengers on 30,000 commercial flights and 35,000 private flights. It also regulates the US aviation industry, certifies the safety of US-built aircraft, and inspects major elements of the US aviation system to ensure safe operation. The replacement value of the supporting IT systems exceeds $25 billion. With so much at stake, the FAA has spent many years evaluating and combining models and methods to establish and monitor IT process improvement. In 1997, it released the integrated Capability Maturity Model (iCMM), which blended three of the Capability Maturity Models—software, systems engineering, and software acquisition— developed by Carnegie Mellon University’s Software Engineering Institute. iCMM version 2, released in 2001, updated and expanded version 1’s set of engineering disciplines to better accommodate the IT processes associated with the agency’s air-traffic-control business, as well as IT system deployment, transition, operation, maintenance, and retirement. It also included some processes that govern IT, such as leadership, strategic planning, and investment decision-making.With this Resources breadth of coverage, we believe the iCMM is the most compre-

Inside

1520-9202/04/$20.00 © 2004 IEEE

hensive model available for improving the performance of an organization that relies on complex IT systems to provide services. The iCMM also has a flexible structure that lets organizations use it to benchmark processes from other process improvement models in terms of either maturity or capability level. Finally, the model offers various appraisal methods so that organizations can understand current practice in relation to iCMM’s best practices or measure process performance characteristics. Lessons have come from many years of iCMM application at FAA, and many have broad application to the IT community. Overall, using one model to cover processes that span many disciplines has clear advantages.

WHY NOT MULTIPLE MODELS? Process improvement is a set of actions an organization takes to change processes so that they more effectively meet business needs and goals. These goals might be to enhance customer satisfaction, create higher quality products and services, lower development and maintenance costs, shorten the time to deliver its products and services, or increase the predictability of product and service development. Since 1991, thousands of IT organizations have improved their software-development processes using the CMM for Software developed at Carnegie Mellon University to capture best practice in key areas such as project management,

Published by the IEEE Computer Society

May ❘ June 2004 IT Pro

43

PERFORMANCE

IMPROVEMENT

Figure 1. A viable process improvement approach (adapted from ISO/IEC TR 15504). Organization’s goals 1 Examine organization’s needs

Process improvement request

Institutionalized improvements

8 Monitor performance

7 Sustain improvement gains

Improvement initiation

Identified scope and priorities 2 Initiate process improvement

6 Confirm the improvement

Preliminary process improvement program plan

Reappraisal request

Implemented improvements

3 Prepare and conduct process appraisal

Analyzed reappraisal results Appraisal results

Appraisal request Current capability

4 Analyze results and derive action plan

quality assurance, and requirements management. The CMM for Software shows an organization how to examine its processes, uncover process weaknesses, and incrementally and systematically adopt best practice.The results of adopting the model have been impressive, with organizations reporting significant increases in their ability to meet needs related to schedule, productivity, product quality, staff morale, and customer satisfaction (J.D. Herbsleb and colleagues,“Software Quality and the Capability Maturity Model,” Comm. ACM, June 1997, pp. 30-40). Despite its success, the CMM for Software is limited to software development, which leaves out much when an organization builds products with substantial hardware in addition to major IT components, such as mobile phones, heart-monitoring equipment, satellites, television, and radar. It also falls short of aiding organizations that create complex service systems—such as overnight package delivery, long-distance phone service, and air traffic control—with many distributed IT components. In the larger process improvement domain, which Figure 1 outlines, the CMM for Software helps only with the needs in step 1 that software alone can address. IT Pro May ❘ June 2004

5 Implement improvements

Approved action plan

Process improvement plan

Industrial benchmarks Practice descriptions from process model

44

Validated improvement results

Target capability profiles

In the mid-1990s, more CMMs began to appear for process improvement in systems engineering, integrated product development, and security engineering. Many organizations—Motorola, Texas Instruments, Lockheed Martin, and the FAA among them—adopted these additional single-discipline CMMs with the aim of improving IT systems as small as mobile phones and as large as military aircraft. Applying these multiple, single-discipline CMMs has proved challenging because their terms and measures differ and their orientation is overwhelmingly toward products, not services.

Lack of standardization The lack of standard terminology among CMMs makes coordination difficult and can leave gaps where disciplines meet. Different design teams, years apart, have produced models that are conceptually similar, but with significantly different terms and structures. For example, most CMMs provide best-practice guidance for performing processes such as configuration or project management, but the Integrated Product Development CMM calls it “base practices,” and the CMM for Software calls it “activities per-

formed.” Depending on the model, the practices fall into process, key process, or focus areas.The models also use different labels for the best-practice guidance for improving process performance: The Systems Engineering CMM calls it “generic practices,” the People CMM calls it “institutionalization practices,” and the Software Acquisition CMM calls it “institutionalization common features.” Perhaps most confusing is the lack of standardization on the measure of process improvement. Some CMMs focus on achieving capability levels in process areas the organization has selected, while others center on maturity levels that reflect process areas that the model defines. Conceptually, both terms denote a set of improvements against which an organization measures itself, but how can the orientations mesh when, say, an interdisciplinary team requires two CMMs? And this is the case more often than not, since interdisciplinary teams commonly perform business processes, and stand-alone models do not foster the development and improvement of integrated processes.

Product, not service, orientation

Table 1. Categories and process areas of the iCMM and associated maturity-level staging. Category

Process area

Maturity level

Management

Integrated enterprise management Project management Risk management Supplier agreement management Integrated teaming

Life cycle

Needs Requirements Design Design implementation Integration Deployment, transition, and disposal Evaluation Operation and support

3 2 3 3 3 2 2 NS*

Support

Outsourcing Alternatives analysis Measurement and analysis Quality assurance and management Configuration management Information management Process definition Process improvement Training Innovation

2 3 2 2 2 NS 3 3 3 5

3 2 3 2 3

*Not staged at a maturity level.

Another drawback is that most CMMs focus on organizations that build products, not those that use products to offer services. Service organizations that deliver packages overnight, provide electrical power, offer long-distance phone service, or control the safe movement of aircraft must acquire, deploy, maintain, operate, and retire complex IT systems. Lacking guidance within existing CMMs for these other business aspects, service organizations often turn to the more general ISO 9001 to guide process improvement or to IEEE/Electronic Industries Alliance (EIA) 12207 for guidance on the IT life cycle. But although both these alternative process guidelines provide insights into quality systems, customer focus, and the IT life cycle, neither offers the depth of process-improvement guidance in a CMM.

INTEGRATED CMM To address the problems of multiple, single-discipline CMMs, the FAA developed the iCMM, which integrates the principles and practices from 10 sources, including ISO 9001, Malcolm Baldridge National Quality Award criteria, international lifecycle and process assessment stan-

dards, and several CMMs. As Table 1 shows, the iCMM provides 23 process areas clustered into management, lifecycle, and support categories, and it groups, or stages, them into maturity levels so that organizations can benchmark process performance with existing CMMs, if they desire. Each process area includes base practices—practices the iCMM considers fundamental to performing that process (not listed in the table). Organizations can perform these processes at various capability levels, where process capability is the process’s ability to achieve a required goal, such as defect rates or speed of product introduction. The higher the capability, the more likely it is that the process will achieve expected results.Table 2 shows the iCMM’s generic practices, which organizations use to enhance the capability of any process. The two types of practices have a useful synergy. Base practices provide guidance on the fundamental practices that any sound process should have. Generic practices tell how to institutionalize and improve the capability of a sound process. A base practice in project management is May ❘ June 2004 IT Pro

45

PERFORMANCE

IMPROVEMENT

Table 2. Capability levels and generic practices for the iCMM. Level

Description

Generic practices

0

Incomplete

None

1

Performed

Identify work scope, perform the process

2

Managed: planned and tracked

Establish organizational policy; document the process; plan the process; provide adequate resources; assign responsibility; ensure skill and knowledge; establish work product requirements; consistently use and manage the process; manage work products; objectively assess process compliance; objectively verify work products; measure process performance; review performance with higher-level management; take corrective action; and coordinate with participants and stakeholders.

3

Defined

Standardize the process; establish and use a defined process; and improve processes.

4

Quantitatively managed

Stabilize process performance

5

Optimizing

Pursue process orientation

to estimate project effort, cost, and other resource requirements. Institutionalizing and improving this base practice might require the assurance that resources are adequate to perform estimation, that the estimation practices are documented, and that people have the skills to carry them out. Generic practices will not ensure that the process can deliver the required performance, but the iCMM calls for establishing process goals, measuring process performance against those goals, analyzing deviations (including root cause analysis), and taking the required corrective action.

pursuing process improvement have consistently reported enhanced productivity, higher quality, increased ability to predict schedules and allocate resources, higher morale, and better communications and teamwork. Along the way, the FAA has learned many lessons, which will benefit nearly any organization that relies on complex IT systems to provide services. Of course, each organization has unique characteristics that affect how it will use these lessons.

LESSONS LEARNED

Process improvement is continuous and requires a consistent effort to tie improvement directly to business needs and goals. The eight-step approach in Figure 1 reinforces both these ideas. At the project level, an organization can use it to identify and carry out action plans to improve a single problem area. For example, one team used it to improve acquisition risk-management practices. At the enterprise level, an organization can improve processes that affect the entire enterprise. For example, one team used the eight-step approach to reengineer the FAA’s internal process for justifying its IT capital investment. Some improvements required cross-functional teams to carry out major projects, possibly revising existing processes or introducing new ones. Other improvements were less ambitious, conducted as part of routine adjustments that process stakeholders identified.The eight-step approach worked in either situation. Step 7, sustain improvement gains, has been particularly valuable in reminding organizational units not to relax or loosen their programs after achieving their goals, possibly falling back into bad habits. Sustainment includes institu-

At the FAA, the iCMM rapidly became the predominant framework for CMM-based improvement. In 1997, the head of the FAA business unit responsible for acquiring air-traffic-control systems targeted several programs to achieve maturity level 2 on the iCMM by December 1999. Other organizations within that business unit gradually joined the effort, as did a separate business unit that maintains and operates the air-traffic-control equipment. The high-level goal of all these efforts was to “realize highquality solutions to Agency and user needs, predictable cost and schedule, and increased productivity.” An agency-wide infrastructure supported the improvement effort, including process experts from across the participating organizations, trainers, appraisers, sample work products, and consultants. To date, 17 FAA organizations have achieved iCMM maturity level 2, three have achieved iCMM maturity level 3, and many other organizations have achieved capability levels 2 or 3 in selected process areas. In addition, several FAA organizations have become ISO 9000 certified. Programs and organizations 46

IT Pro May ❘ June 2004

Use the ISO/IEC 15504 process improvement approach

tionalization—spreading benefits by rolling out validated gains across organizations. Once a team validated the improved practices for acquisition risk-management, for example, other organizations were able to share the benefits. The same institutionalization occurred with the improved processes for developing and fielding new system releases.

Explicitly tie goals to performance

All organizations modify their processes over time, if only in response to crises rather than anticipation. When done well, modifications improve organizational performance. On the other hand, if an organization simply focuses on improvement without an explicit tie to business performance, the effort will lack focus, and managers will lose interest. Because the iCMM provides 23 process areas and defines six increasing levels of process capability, Reengineer and institutionalize A process performs poorly primarily for one of two rea- improvement teams can select just the process areas and sons. First, the process steps might prove unsuitable for the capability levels most relevant to their business. For examintended performance. The now discredited big-bang ple, an FAA research organization focused on improving approach to software development assumed that, at the their innovation processes, an acquisition team concenproject outset, a team could with reasonable confidence trated on deployment, and a testing organization aimed to better the capability of its evaluadefine cost, schedule, and requireUsing a single model, tion processes. ments for large programs. Broad experience at the FAA and elsedisjoint parts of an where has shown that the big-bang Adopt a multidisciplinary process almost inevitably fails no organization will begin model matter how hard the team tries. A primary benefit of using a sinto talk. Analytic techniques simply cannot gle model across disciplines is reliably predict cost and schedule greater efficiency because everyone years in advance and derive stable requirements with any involved in the process improvement speaks the same precision. vocabulary; benefits from the same training, planning, temSecond, the team might not be able to fully execute a plates, and documentation; and can more easily compare sound process.A team untrained in testing techniques will results. not be able to develop and execute a satisfactory test regAnother benefit is that disjoint parts of an organization imen—even if the testing process is world class. will begin to talk. Issues that might otherwise have gone The organization must understand what is causing poor unnoticed will come to the forefront.The result is a greater performance—is it inherent flaws in the process, inade- appreciation for how various parts function, a rising interquate execution of a sound process, or some combination? est in group problem-solving, and a general increase in The FAA found that reengineering techniques offer an cooperation across the organization.This harmony is both effective way to establish a new process with the needed vertical—since the iCMM covers management practices characteristics. Reengineering includes a root-cause analy- at strategic, project, and team levels—and horizontal— sis of why a process is performing below standard; pro- since practices span the entire product or service life cycle poses a new process to address the substandard per- from conception through disposal. Executives, managers, formance, which could include the best practices in the system engineers, software engineers, contracting officers, iCMM; and tries out the new process in a limited applica- and supporting practitioners develop an appreciation of tion to ensure that it works adequately. If the trial is suc- their respective roles and clarify how they interrelate. cessful, the process is ready for institutionalization. Teams learn to respect and rely on often overlooked yet It is relatively easy to institutionalize generic practices critical roles played by, say, configuration management offibecause they are based on commonsense practice familiar cers and quality assurance practitioners. to many stakeholders.As one FAA executive put it,“[The use of generic practices] is Management 101.” Moreover, Adopt a full-lifecycle model once the generic-practice concepts become institutionalStrong business performance requires strong execution ized, it becomes quite natural to apply them even to across the entire life cycle. For a business that provides a processes such as financial management, which the iCMM service, the focus must be much broader than IT systems does not address directly. For example, the CIO’s business development. A process improvement framework that office is using the iCMM to improve budget planning and addresses development in depth but ignores deployment, execution. Planning a process, assigning roles and respon- transition, and operation is not suitable for the FAA or sibilities, providing resources, managing work products, and any other organization providing a service.The FAA often measuring performance against the plan become intuitive. spends several years fully deploying a system after buildOnce an organization recognizes and accepts the value of ing the first engineering article. Because no two airports the generic practices, this becomes a cornerstone in build- or other air-traffic facilities are identical, much of the ing a corporate culture of continuous improvement. deployment is to adapt each system to the facility’s unique May ❘ June 2004 IT Pro

47

PERFORMANCE

IMPROVEMENT

Resources CMMs ➤ FAA Integrated Process Group (http://www.faa.gov/ipg): Information on the iCMM, its appraisal method, mapping tables, and related documents and publications. ➤ Software Engineering Institute (http://www.sei.cmu.edu): Information on Carnegie Mellon University’s CMMs and the CMM Integration model. ➤ Guidelines for Using FAA-iCMM v2.0 and ISO 9001:2000 in Process Improvement, L. Ibrahim and C. Wells, FAA, 2004; http://www.faa.gov/ipg. ➤ “Integrating Beyond CMMs,” L. Ibrahim, Proc. 12th Int’l Symp. Int’l Council on Systems Engineering (Incose), Incose, 2002.

tions pursuing iCMM-based process improvement might also have a business objective to achieve ISO 9001 certification; others that are already ISO 9001 certified might have additional business goals that iCMM can support. FAA’s experience has highlighted the need to develop explicit guidance on how to use the iCMM for such multiple purposes. We are currently piloting a project in collaboration with ISO 9001 auditors to provide the means by which organizations can attain both ISO certification and iCMM goals from a single appraisal.

Know where you are

Baselining current process performance is essential to improving it.As part of baselining— the beginning of step 3 in the ISO/IEC 15504 approach in Figure 1—organizations take two performance measures: how well the process Standards performs with respect to traditional measures ➤ IEEE/EIA 12207.0-1996, Industry Implementation of Interof cost, speed, and effectiveness and how well national Standard ISO/IEC 12207:1995, Standard for Inforthe process incorporates best practice. An mation Technology– Software lifecycle processes, IEEE, appraisal at the start of the process improve1998. ment effort establishes the baseline, and the ➤ ISO/IEC TR 15504:1998E, Information Technology–Softorganization should periodically conduct ware Process Assessment, Part 7: Guidelines for Software appraisals to understand how well it has instituProcess Improvement, Int’l Organization for Standardizationalized the improved process.All CMMs protion and the Int’l Electrotechnical Committee, 1998. vide appraisal methods to determine which best ➤ ISO 9001:2000(E), Quality Management Systems – Requirepractices a process includes and to determine if ments, Int’l Organization for Standardization, 3rd ed., 2000. everyone is following them. ➤ ISO Online (http://www.iso.ch): Information on ISO/IEC The FAA has developed a variety of appraisal standards. methods to use with the iCMM, starting with a questionnaire-based technique that takes about a day or so.A full appraisal method can take up characteristics, such as runway configurations and the facil- to two weeks and involve interviews with dozens of staff, ity site’s weather and topology. Service must continue dur- an extensive document review, and a team of independent ing the new system’s deployment, which means the old and appraisers. In addition to measuring process capability, the new systems must run side by side for some time.Without iCMM has an appraisal method to measure the usefulness a plan for this tandem operation, smooth coordination and cost-effectiveness of process performance results.The would be difficult, if not impossible. different methods have different objectives, and the FAA has used them to launch improvement efforts, uncover major issues, track status, monitor progress, confirm fullExploit simultaneous improvement Organizations must recognize the efficiencies they gain appraisal readiness, and formally benchmark capabilitywhen using a single integrated model rather than individ- or maturity-level achievements. For example, when we wrote this article, one project with ual source standards and models in tandem. Improvements using the iCMM simultaneously yield improvements against a substantial development budget was halfway through its all its integrated sources, and detailed mapping tables indi- development life cycle and had been experiencing schedcate the alignment.Achieving maturity level 2 on the iCMM, ule slippage, requirements creep, and other problems. It for example, aligns with achieving maturity level 2 on all its had neither a formal process improvement program nor source models that define the maturity levels in the acqui- an appraisal to understand how many best practices its processes contained. The project team agreed to undergo sition, software, and systems-engineering disciplines. Because the iCMM combines several international stan- a medium-weight appraisal of the problematic process dards, including ISO 9001, many organizations have areas with the aim of addressing discovered deficiencies. expressed interest in using the iCMM to both enhance The appraisal method selected will produce almost all the process capability and become certified. Some organiza- desired insights at a very modest cost. Because a variety 48

IT Pro May ❘ June 2004

of appraisal methods were available, the FAA could easily address these problems.

Establish an integrated infrastructure Integrated process improvement requires an integrated infrastructure. The FAA’s process improvement infrastructure includes • sponsors at several levels, starting with the CIO and business unit heads; • process groups at several levels, starting with the corporate integrated-process group; • corporate working groups, including cross-organizational teams for iCMM evolution, measurement, training, communication, process asset library, and appraisal; and • process action teams. Perhaps most important is the need to secure top management’s sponsorship and commitment. An enterprisewide process group, staffed with executives and senior technical people who are widely respected, motivated, empowered, and persistent, should lead, advocate, and coordinate the effort.

rity, as well as guidance for iCMM use in planning, developing, applying, and implementing an enterprise architecture. Since the iCMM appeared in 1997, Carnegie Mellon University has published the CMM Integration, another CMM that integrates several single-use CMMs. It also eliminates weaknesses of single-use CMMs, but is currently more narrowly focused on IT system development and therefore excludes acquisition and the lifecycle phases of deployment, transition, and operations. It also does not include strategic planning and leadership. As the CMM Integration model evolves, it could become as powerful a framework for improving processes as the iCMM. We believe the iCMM is flexible enough for selective use and broad enough to encompass the most commonly occurring processes in a typical enterprise. Using a single model that draws together widely recognized standards across government and industry will provide a consistent approach to achieving performance excellence, and the FAA will continue to evolve the iCMM toward that vision. ■

Linda Ibrahim is chief engineer for process improvement at the US Federal Aviation Administration. Contact her at [email protected].

Ensure appropriate training At the FAA, a corporate training group manages iCMMrelated training. Staff provides training on using the model and appraisal method and on how to do process improvement in general. Domain- or discipline-specific training, such as that in requirements engineering, project management, configuration management, and quality assurance, is the responsibility of those in the FAA with the relevant skills. Several FAA sites offer training, and special workshops are available upon request. Since the iCMM is not specific to the FAA, model training is regularly available to the public.

Arthur Pyster is deputy assistant administrator for information services and deputy chief information officer at the US Federal Aviation Administration. Contact him at arthur. [email protected].

IEEE Computer Society members today! online Join

a Not

Over the next five years, the FAA is committed to reducing the already very low rate of fatal aircraft accidents, meeting the projected demand for more air travel, and increasing the safety and capacity of the global civil aerospace system. It will do so while strengthening the agency’s leadership, improving its workforce training, enhancing cost-control measures, and improving decision-making through more reliable data. To meet this ambitious agenda, the FAA must continuously improve the performance of the processes that guide the acquisition, deployment, transition, operation, maintenance, and retirement of its IT systems. Years of using the iCMM have demonstrated the value of a single integrated enterprise-wide process-improvement model, but there is always room for enhancement. Planned additions include standards-based best practices in safety and secu-

member?

FIVE-YEAR PLAN

save

25

%

on all conferences sponsored by the IEEE Computer Society www.computer.org/join May ❘ June 2004 IT Pro

49