AUTHOR: TITLE: SOURCE: COPYRIGHT:
Kirsten M. Rosacker; David L. Olson An Empirical Assessment of IT Project Selection and Evaluation Methods in State Government Project Management Journal 39 no1 49-58 Mr 2008 The magazine publisher is the copyright holder of this article and it is reproduced with permission. Further reproduction of this article in violation of the copyright is prohibited.
ABSTRACT This study uses a survey of U.S. state government information technology [IT] project management practitioners to investigate the utilization of IT project selection and evaluation methodologies -- financial and qualitative -- and to assess the empirical relationship between the chosen methods and several measures of perceived project success. The analysis presents evidence that financial project selection and evaluation methodologies appear to be important in obtaining better control over project costs. KEYWORDS: project selection; information systems projects; government information systems INTRODUCTION Information technology (IT) is clearly valued in all business environments, as evidenced by both public- and private-sector organizations investing in IT projects at an everincreasing rate. Such projects, however, are noted for their difficulty in successfully balancing time, budget, and quality requirements (Keil, 1995; Keil, Mann, & Rai, 2000; Olson, 2004). A universally preferred method of ranking for selection and evaluation among various IT projects does not appear to exist from among the standard methods -financial and qualitative -- that have been specified and assessed in the literature. Indeed, the selection and use of project methodologies seems to be contingent; upon the personal preferences of the evaluators and the operating environment of the organization and/or industry. Prior research related to IT project selection and evaluation techniques has largely concentrated on the private sector, although some studies have considered the public sector (Bozeman & Kingsley, 1998). Study of the public sector is important for at least two reasons. First, the United States public sector (federal, state, counties, and cities) represents the largest procurer of IT resources and services in the world. Second, a primary motivating factor for private-sector organizations in their assessments of investment opportunities is the desire for profit. Public-sector entities, on the other hand, exhibit a preference for cost containment and may not have a direct or indirect means to focus on bottom-line considerations. Therefore, methods that emphasize profits may not be readily applicable to, or desirable for, public-sector decision making. While IT is ubiquitous within public-sector organizations (Sonde, 2004), the successful adoption, implementation, and expansion of IT has been historically constrained. For example, the Internal Revenue Service still employs 1960s technology to manage the reporting and collection of tax payments -- the primary revenue source supporting federal government operations. Social service environments that are responsible for distributing welfare transfer payments to lower-income individuals and families are primarily based in outdated technology. Their systems are represented by 1970s-cra mainframe computer and COBOL programming that was a constant for that time. And, adding to the reluctance to change, the Standish Group (2004 and subsequent) reports that 80% of all private- and public-sector IT projects ultimately fail. Given the magnitude of IT spending and the unique operational and environmental characteristics that can be found in public-sector organizations, as well as the desire of taxpayers to ensure that government officials spend funds in a rational, efficient, and effective manner, the topic of public-sector IT project selection and evaluation methodologies warrants investigation. It would be helpful to all interested parties if a set of specific and controllable factors were identified that contribute to the successful implementation of information systems (IS) within state governments. The identification and empirical validation of such factors, commonly referred to as critical success factors (CSFs), within the context of state government IS projects provides the motivation and focus for this study (Jiang, Klein, & Balloun, 1996). PURPOSE AND SIGNIFICANCE OF RESEARCH The primary purpose of this paper is to identify and describe the project selection and evaluation techniques and rationales employed by state governments in ranking and ultimately selecting among multiple IT opportunities. This is done through a survey of state government IT project managers, contacted through directors of state IT departments. A number of studies assessing IS implementation project success factors (Finch, 2003; Fitzgerald, 1998; Pinto & Slevin, 1989) have been reported in the literature. The vast majority of these studies deal with private-sector organizations or a mix of public- and private-sector organizations. Although there are many similarities between private-and public-sector organizations, it is reasonable to infer that these organizations may deal with information system project management differently (Bretschneider, 1990; Caudle, Gorr, & Newcomer, 1991). There are several substantive reasons why a study dealing specifically with state government IS implementation project success factors is important and must be undertaken. First, state governments spend a significant amount of money on IT and will continue to do so into the future. State and local governments spent US $39.9 billion and US $40.4 billion on IT. Additionally, this amount is expected to increase to US $40.7 billion and US $43 billion in 2004 and 2006, respectively (Welsh, 2004). Second, IT project implementation is very problematic in any environment. Merely 20% of all private- and public-sector IT projects are considered successful (Standish Croup, 2004). State government agencies encounter "substantial difficulties in delivering IT initiatives on time, on budget, and with the desired features." More specifically,
"the average state experienced overruns, delays, and shortfalls by approximately 45%. Approximately 80% also experienced feature shortfalls. Roughly 90% of the projects experienced cost overruns and schedule delays of 25% or more" (Brown, 2003). Third, despite the widespread use and increasing importance of IS in state government organizations, prior research regarding IS implementation project success has focused on the private sector. The findings from these studies may not hold for state government IS implementation projects. Bajjaly (1999, p. 46) stated that "implementing emerging IS can be especially risky for public agencies because the CSFs have neither been researched sufficiently nor documented adequately within public sector agencies." Public-sector managers having responsibility for the implementation of IS cannot assume that the set of determinants for IS success in private-sector organizations are equally important for public-sector organizations. Even if one normatively believes this to be true, the contention must be empirically assessed. This study reports the results of a survey of the project selection and evaluation methods utilized in state government TT projects. It compares the current findings with past research findings on IT project selection methodologies, all of which focused on the private sector. Additionally, the empirical relationship between various selection methods and several metrics of project success are evaluated. PREVIOUS RESEARCH EFFORTS DIFFERENCES BETWEEN PRIVATE- AND PUBLIC-SECTOR IT PROJECT SELECTION Bozeman and Bretschneider (1986) argued that public- and private-sector organizations are similar in many ways but also quite different in several significant respects. They asserted that public-sector organizations are dominated by concerns of accountability, openness, and "representationalness." Public-sector organizations are extremely risk-averse, subject to divided authority, experience short-term rather than strategic budgets, and utilize a highly regulated procurement process. These characteristics serve to hamper a public-sector organization's ability to implement and maintain contemporary information systems. Bozeman and Bretschneider provided a framework for studying public-sector management information systems (MIS) that are based on the identified differences between public- and private-sector organizations. However, their framework was not used to conduct an empirical assessment. Several studies survey the differences between public- and private-sector information management procedures. Bretschneider (1990) conducted a survey of public-sector information executives. His results indicated that public-sector organizations exhibited "greater interdependencies, leading, at least in part, to increased accountability, procedural delays, and red tape" (p. 543). Economic issues are more significant within private-sector organizations than in public-sector organizations. Caudle et al. (1991) utilized a survey to identify IS priorities of both public- and private-sector organizations. Their results indicate that (1) middle-level (rather than top-level) public managers are crucial for IT development; (2) small government agencies arc more interested in IT transfer than large ones; (3) governments with a lot of red tape tend to have flexible IT; and (4) local government IT issues are driven by transaction processing, while state and federal governments have IT systems that are more suitable for their oversight mission. Because the findings varied by level of management and government, it was difficult for the researchers to formulate general statements regarding public- and private-sector information systems. Any conclusions needed to be stated contingently based on the level of management and level of government. Bozeman and Kingsley (1998) argued that politics disrupts the long-range planning necessary for information resource planning. Nutt (1999) concluded that "most public organizations do not have the luxury of keeping strategic decisions secret. Sunshine laws often force the conduct of business into the open... Even when sunshine laws do not apply, mechanisms of accountability and oversight make all actions in public organizations, even contingency plans or hypothetical scenarios, subject to review and interpretation by outsiders" (p. 312). IT PROJECT SELECTION METHODOLOGIES Organizations appraise their IT investments for several reasons, including to justify their investments, to enable organizations to decide between alternative projects, to control IT expenditures, to improve the investment selection process, and to facilitate project management (Ballantine & Stray, 1998; Farbey, Land, & Targett, 1992; Ginzberg & Zmud, 1988). Two broad categories of IT project appraisal techniques commonly used in evaluating IT investment opportunities can be identified: financial and qualitative (Bannister & Bemenyi, 2000). This may be because while the selection process is recognized as important, information systems projects involve group-oriented activities subject to the benefits and problems of group dynamics, interactions, coordination, and communication (Omitaomu & Badiru, 2007). Thus, qualitative approaches often override quantitative approaches, which may require more detailed data than is reasonably available. FINANCIAL PROJECT SELECTION AND EVALUATION METHODS Several methodologies exist and have been employed in evaluating IT investments based on financial criteria including cost benefit analysis (CBA), budgetary constraint, payback, net present value (NPV), and internal rate of return (IRR) methods. CBA involves determining the anticipated costs and benefits of investing in an IT project. In general, for an IT project to be acceptable under this technique, its benefits should exceed its costs. When utilizing the budgetary constraint method, an organization will simply select the number of projects that can be entertained within its current budget -- an extremely simplistic methodology. The payback approach to IT investment analysis involves estimating the time required to recover the initial investment and ranking the projects based on this criterion. Advanced financial analysis techniques view an IT project as a series of cash outflows and future cash inflows composed of two important components: (1) recovery of the
initial IT investment and (2) income (Fess & Warren, 1993). The NPV method, also referred to as the discounted cash flow method, employs financial evaluation formulas to compute the NPV for all cash flows, both inflows and outflows, that are expected to accompany a potential IT investment based upon an organization's expected minimum rate of return. Use of NPV in IS projects has been addressed in the literature (Flaig, 2005; Mullen & Donnelly, 2006). The NPV method evaluates the current value of estimated future cash flows on the assumption that future benefits are subject to this discount factor (e.g., cost of money). IRR is the discount rate that yields a zero NPV for the project under consideration. An organization's management will determine the minimum acceptable IRR for an IT project -- generally referred to as the hurdle rate. More advanced financial analysis variants have been proposed (Liesiö, Mild, & Salo, 2007; Patton & Shechet, 2007; Yu, Flett, & Bowers, 2005) in the literature. Organizations that exercise a strict financial discipline often use formal financial methods (Farbey et al., 1992). Project risk is widely recognized as important, indicated in part by typical failure rates previously cited. Chapman, Ward, and Klein (2006) proposed a procedure considering risk management in project portfolios. Monte Carlo simulation is another tool to aid risk analysis, and was applied by Paisittanand and Olson (2006) in an IT project. Consideration of risk and other criteria in addition to expected cost make multicriteria methods interesting. Gabriel, Kumar, Ordonez, and Nasserian (2006) combined analytic hierarchy process with simulation for multiobjective project selection. Mohanty, Agarwal, Choudhury, and Tiwari (2005] and Cheng and Li (2005) proposed analytic network process, considering nonlinear feedback models, for R&D project selection analysis. Different forms of multiple attribute analysis (MAUT) were proposed by Duarte and Reis (2006) in their project evaluation system, and by Kulak, Kahraman, Öztaysi, and Tanyaçs; (2005) in their project selection system using fuzzy modeling. However, while these approaches are interesting, they are not widely used in practice. None of the subjects surveyed indicated that they used such advanced techniques. The MIS literature describes several significant disadvantages to utilizing financial appraisal techniques that may substantially limit their application to IT project selection. Ballantine and Stray (1998) argued that IT is fundamentally "different from other types of capital investments, and as a result, financial techniques which have historically been used to appraise capital investments are inappropriate for appraising IT investments." These financial techniques often overlook intangible benefits associated with IT investments, thereby understating the project's true value (Schell, 1986). Hochstrasser (1992) pointed out that financial appraisal techniques emphasize profit, which is unsuitable for many IT investments that could be undertaken to improve customer support and/or offer better market information. Finally, these financial methods are based in capital budgeting theory that makes assumptions regarding cash flows and discount rates that are merely estimates (Bacon, 1992). Although many more advanced financial analytic methods exist, a pilot study did not find their presence evidenced in state government IT project selection. As a result, these methods were not included in the survey sent to state government IT project managers. Our purpose here is descriptive, not normative. Therefore, the survey was based on techniques reported actually utilized by the respondents. QUALITATIVE PROJECT SELECTION AND EVALUATION METHODS Qualitative attributes represent business and/or project characteristics that can be identified but cannot be easily quantified (Klein & Beck, 1987). Qualitative characteristics include instinct, probability of completion, existence of a project champion (top management support), and mandatory requirements. The MIS literature provides evidence that managers occasionally base IT investment decisions on instinct (Powell, 1992), acts of faith (Dietz & Renkema, 1995; Farbey et al., 1992), and/or blind faith (Weill, 1990). Instinct is a subtle form of reasoning that takes into account how the world really is rather than merely "relying on what the spreadsheet says" (Bannister & Remenyi, 2000, p. 237). Many IT projects will not proceed without the presence of a project champion (Farbey et al., 1992). The project champion, generally a member of top management, is someone who has the influence to ensure that the project has sufficient priority to enable success. This individual is also responsible for providing the funding and staffing resources to ensure the project's success. Additionally, IT projects will only be undertaken that have a high probability of reaching completion, A legal requirement, demands of an external customer or supplier, and competitive pressures may mandate an organization to undertake an investment in IT (Fitzgerald, 1998). IT PROJECT SELECTION IN PRACTICE A number of methods exist to evaluate project proposals, either from the perspective of selecting the best option available, designing an ideal option, or rank-ordering options. There have been a number of studies examining the use of IT project selection. Bacon (1992) surveyed senior executives in 80 organizations about how they allocated strategic IS technology resources. Executives were presented with 15 criteria, including six financial, six managerial, and three development criteria. The criteria used most were the management criteria of supporting business objectives and support of management decision making (88%). Other criteria with high reported percentages of use were technical/system requirements (development criterion, 79%) and legal/government requirements (management criterion, 71%). None of the subjects reported that any particular criterion was used for all projects considered. Subjects were also asked to rate the overall value of the 15 criteria. Support of business objectives was rated first, with internal rate of return and NPV rated next, followed by payback. Powell (1992) reviewed studies of computer system investment appraisal, which he divided into objective and subjective evaluation methods. Objective methods seek quantified values for project proposals. Subjective methods acknowledge the frailty of such value estimates and emphasize attitudes and opinions. Powell concluded that formal objective techniques are used in only a small number of cases, and where they are employed it is often for after-the-fact justification. Other studies were found to imply a growing use of cost-benefit analysis, more often in larger firms, for specific classes of IS/IT projects. Tam (1992) surveyed 184 senior MIS personnel and management executives, finding that capital budgeting had little impact on IS investment. Simple techniques such as payback were preferred over more complex quantitative models. The use of techniques varied widely within firms, with particular techniques not used for every proposal. User groups generally thought that financial capital budgeting techniques were useful, but nonusing groups were less convinced. Problems with capital budgeting techniques
cited most often were difficulties in estimating returns, followed by difficulties in estimating costs. Hinton and Kaye (1996) surveyed 50 members of a professional organization whose members were responsible for appraising key organizational investments. Methods used for IT projects were appraised and compared to projects in other areas [operations, marketing, and training). Treatment of a project as a capital investment involves costbenefit analysis to establish profitability. Treatment as a revenue-related project does not require cost-benefit analysis, as the project is expected to foster key organizational goals, and the benefits are recognized as being difficult to measure accurately. Projects involving investment in operations and IT were usually treated with some form of costbenefit analysis. Investments in training and marketing were usually treated as revenue-related projects, and thus cost-benefit analysis was for the most part foregone. Cabral-Cardoso and Payne (1996) surveyed 152 U.K. research and development decision makers about their use of selection techniques. Financial techniques were used by about two-thirds of those surveyed, with the simpler payback method used most often. About one-third of those surveyed used subjective techniques (or techniques combining subjective and objective features) that included checklists, project profiles, and multiple criteria analysis. Jiang and Klein (1999) explored the criteria perceived as important by 88 IS managers in the context of strategic relevance. They found three groups of evaluation criteria, with financial criteria and organizational needs grouped together, technical IS matters grouped with user need and management support, and factors such as customer/supplier requirements, industry standards, legal requirements, and response to competition as the third group. RESEARCH METHODOLOGY This research uses a survey methodology, To identify and gain permission to contact state government IS implementation project managers, a letter was sent to the chief information officer (CIO) of several states. This letter requested names and contact information for IS project managers during the past 7 years. Contact information for the CIOs was provided by the National Association of Stale Government Chief Information Officers. Sixteen states responded, for a total of useable 144 responses as presented in Table 1. PROJECT SELECTION AND EVALUATION METHODS Each IS project manager was sent a survey instrument via e-mail. Subjects were asked if the selection of their project had included the following methods: CBA, NTV/IRR, payback, budget constraint, probability of successful completion, legal requirements, driven by top management support, and/or subjective assessment. The 144 responses provided the basis for the proportions reported in Table 2. In Table 2, the proportion of users in private firms reported by others is given. The proportion obtained from the; survey is given, along with a 0.95 confidence interval based on the proportion confidence interval formula (Evans & Olson, 2003): CI = p +/- z[subalpha/2][root]p × (1-p)/n The confidence interval in Table 2 is based on an alpha of 0.95. A confidence level of 0.99 was also tested to evaluate significance of the difference between the survey proportion and the proportions reported by other reports. The differences between methods used reported by government project managers in our survey were all significantly lower than the proportions of method use reported by prior studies. Cost/benefit analysis was the closest, with the confidence interval at 0.95 not quite overlapping the reported 0.63 in the Cabral-Cardoso and Payne (1996) study (conversely, significantly larger than the usage of 0.15 reported by Tam [1992]). Cost/benefit studies are Time-consuming but not very accurate, as they require estimated and assumed costs that may not be nearly as accurate as implied by the method. But they are often required, especially in governmental studies. More bureaucratic private organizations may require them as well (and they are implied as needed in higher levels of Capability Maturity Model Integration 1CMM1; 1993], which require such formal project adoption methodologies). Although the use of CBA is lower in our survey, it is the highest reported quantitative method, reflecting in part that it is the method that is most well known. The other quantitative methods appear to be used less in government projects than in other environments. NPV and associated IRR are not often appropriate in a government setting. Even so, 23% of the responses reported using one or the other method. This is, as would be expected, lower than the rate of use reported in other sectors (Bacon, 1992; Cabral-Cardoso & Payne, 1996; Jiang & Klein, 1999). NPV/IRR methods require even more assumptions about gains and costs than does CBA. Payback is a much simpler financial analytic method. Payback had the lowest reported rate of use in the survey of governmental project managers. Conversely it was relatively widely used in other sector reports (Bacon, 1992; Cabral-Cardoso & Payne, 1996; Powell, 1992). Cost savings arc expected to he important in the government sector but are usually expressed in more formal cost/benefit analyses than the faster, less precise payback method. The method of budget constraint has been reported as a fairly widely used method (Bacon, 1992). This approach is often useful in governmental operations, especially in agencies such as the Corps of Engineers, where it is considered a wise policy to have a number of candidate proposals "on the shelf" so that they can be quickly forwarded for approval as end-of-fiscal-year budgets loosen up. However, in our survey, this approach was reported used by only 25% of the sample. The assessment of the probability of project completion is important in IT projects, which have relatively high levels of uncertainty with respect to feasibility. The 46% rate reported by Bacon is significantly higher than the 29% found in our survey, but both results indicate that this factor is considered important. Many governmental IT projects are required by federal regulations or by state legislation. This is expected to be much more important than in private concerns, where the equivalent driver would be top management. Both of these methods (or motivations) were reported by about 67% of the cases in the governmental survey. Often more subjective methods are used to select IT projects. Use of subjectivity was reported (Jiang & Klein, 1999; Tam, 1992), with the proportional usage rate as high as 88% (Bacon, 1992). In our survey, this figure was much lower, at 26%. This number may indicate less willingness to base IT project selection on subjectivity in government
operations, where justification is required by public scrutiny. Finally, top management support was reported in only one of the prior studies (Jiang & Klein, 1999), without a proportional rate of use to compare with. In our survey, this was the method or basis of project selection that had the highest reported rate of use. In either the private or the government sector, it has long been conventional wisdom that without top management support, IT projects die. SUCCESS METRICS A successful IS implementation project, for purposes of this study, is defined as having satisfied objective measures articulated and developed by Belout and Gauvreau (2004). Bach construct was evaluated using a 5-point Likert scale against a statement representing the success construct. First, the project must have been implemented in a timely manner. This was assessed with the following statement: "The project will come in on time." Second, the project's overall cost must have been within budget constraints. This was evaluated with the following statement: "The project cost objectives will be met." Third, the implemented solution must contain the features and functionality requested by the end-user. Two statements were used to develop empirical measures for this construct covering intended use ("The project will be used by its intended clients") and positive impact ("The project will have a positive impact on those who make use of it"). Finally, overall project success expectations were assessed with the following statement: "All things considered, this project will be a success." The results of these responses were compared with methods used through A NOVA. Table 3 shows results obtained. SUCCESS METRIC #1 - TIME Two methods were significantly associated with the time dimension of project success. Mandatory projects (requirements) had highly significant association with project time success. This could be interpreted as concluding that delays were not incurred by wasted time with detailed selection evaluation when the project was known to be required, a spirit that evidently continued on through various project phases. The other significant relationship with time success was the use of subjective assessment. This relationship was negative, indicating that basing project selection on what were perceived by project managers to be subjective means was statistically associated with project complications that led to late projects. The payback method was not significantly associated with project time performance, but we note that its use was usually found in projects that tended to be late. SUCCESS METRIC #2 - COST The use of budget constraints as a basis for project selection was very significantly related to positive project cost performance (or expectations). The NPV methodology was not widely used as indicated in Table 1, but when it was used, those projects tended to perform well on cost. There were no other significant relationships between cost performance and methodology but those projects viewed as adopted based upon subjective means bad a negative correlation with project cost performance expectations. SUCCESS METRIC #3 - USE None of the project selection methods was significantly associated with client use expectations. Three of the methods had negative (although insignificant] associations with client use -- CBA, payback, and top management support as the basis for project selection. It is possible to conclude that project selection and evaluation method has little to do with expected client use. SUCCESS METRIC #4 - IMPACT Only subjective bases for project selection had significance with expected project impact (an assessment of whether the project improved operations or not), and that was weak. A reasonable conclusion is that project selection method has little to do with expected project impact. SUCCESS METRIC #5 - OVERALL SUCCESS Overall expected project success was found weakly associated with probability assessment (which was rarely used) and required projects. There was an insignificant but negative relationship between the use of the payback method and overall expected project success. We would expect required projects to have clear focus and attention, and thus lead to better chances of success. We did not expect the use of probability assessment methods to be widespread in government IT project selection, and it wasn't. But it is interesting that when it was used, there was positive association with expected project outcome. CONCLUSIONS IT is an important tool for the efficient operation of state governments. Selection of IT projects is important in any environment including state government, there have been prior studies about the use of IT project selection techniques in general IT environments. This study reports survey results of IT project managers in state governments and compares them with prior studies. There were some differences detected in IT project selection between state government environments and prior studies. Cost benefit analysis was widely reported in both, although the state government survey found significantly less use of this method than reported in general environments. CBA was not found to be significantly associated with
any of the five measures of project success considered in this study. Other formal methods were found to be less common in the state government survey. This is probably due to the need for formal justification in state government environments, also indicated by the much lower use of subjective project assessment. However, the use of NPV, probably the most formal evaluation technique, was lower in state governments. This is probably because state governments are not as driven by the need for profitability. NPV was significantly associated with expected project cost success. The use of payback, a quick simplification of financial analysis, was much less in state government than in other sectors, probably due to the focus on effectiveness in accomplishing required tasks rather than on profit impact. Payback was not significantly associated with any success measure used and, in fact, had negative correlations with three of these success measures. Budget constraints as the basis for project selection were highly associated with expected project cost success. Probability assessment techniques were not widely used in either environment but had the highest association with overall expected project success. Although not assessed in the other sampling surveys reviewed, top management support is recognized as an important factor in project selection success. There are two factors that showed up as quite important in state government IT project selection that were not surveyed by the prior studies. These were required projects and top management support. Projects that are required obviously will be adopted. This factor has always been present in state government environments. With the passage of legislation such as Sarbanes-Oxley, which increased reporting requirements, private IT environments will probably see an increase in the importance of required IT projects. Requirements as the basis for project selection were most associated with expected project time success, and also were weakly associated with expected overall project success. Top management support has been a factor of importance in all project environments, and is expected to continue to be important. Top management support was cited as a factor in selection more than any other technique. However, there was no significant association with top management as the basis for project selection and expected project success. A subjective base for project selection was intended as a catchall method for lack of method. This category was not widely reported as used but was significantly associated with negative expected project time performance, and while not significant, had a negative correlation with expected project cost performance. There was a weakly significant positive relationship detected with project impact, but we would view this as spurious. Overall conclusions are that financial methodologies can be important to get better control over project cost. Probability assessment methods (such as simulation) are not widely used but appear to have value in overall project success. The payback method appears inappropriate (and not widely used) in the government environment. ADDED MATERIAL Kirsten M. Rosacker, assistant professor of business at the University of South Dakota, received her BS in accounting from Mankato State University, her MS in taxation from The University of Akron, and her PhD in business from the University of Nebraska-Lincoln. Her research interests include investigating the impact of applying business best practices within a government setting. David L. Olson is the James & H. K. Stuart Chancellors Distinguished Chair at the University of Nebraska. He has published research in more than 100 refereed journal articles, primarily on the topic of multiple objective decision making. He teaches in the management information systems, management science, and operations management areas. He has authored or coauthored 20 books, including Decision Aids for Selection Problems, Managerial Issues of Enterprise Resource Planning Systems, Introduction to Business Data Mining, and Enterprise Risk Management. He is a Fellow of the Decision Sciences Institute. Table 1: State government respondents and project selection/evaluations methods. U.S. State Assessment CT 0 FL 0 ID 1 IA 3 KS 1 LA 2 MI 1 MN 7
Budget Probbility of Constraint Completion
Subject
Count
CBA
NPV/IRR
Payback
Legal Reqmts
Top Mgmt
1
0
0
0
0
1
1
1
6
6
0
0
2
3
6
5
1
0
0
0
0
0
1
1
7
3
4
1
5
2
4
5
8
6
2
2
2
1
4
8
2
1
1
1
1
2
0
2
3
3
1
0
1
1
2
3
19
10
5
6
4
6
13
13
MS 13 NE 2 NV 0 ND 7 OH 0 OR 1 UT 0 WY 0 Totals 38
55
23
12
6
10
15
39
34
7
2
1
1
0
2
4
3
2
1
0
0
1
0
2
1
14
9
5
3
5
5
10
8
1
1
0
1
0
0
0
0
11
7
1
1
1
1
7
6
1
1
0
1
0
0
0
1
6
5
1
0
0
2
3
6
144
78
33
23
32
41
96
97
Table 2: Comparison of method use by proportion.
Method CBA NPV/IRR Payback Budget constraint Probability of completion Requirements Top management support Subjective
Prior Reports 0.63 0.40-0.54 0.61-0.68 0.68 0.46 None given None given 0.88
Survey (N = 156) 0.542 0.229 0.160 0.222 0.285 0.667 0.674 0.264
0.95 Cl Significance of Survey of Difference 0.460-0.623 0.99 0.161-0.298 0.99+ 0.100-0.220 0.99+ 0.154-0.290 0.99+ 0.211-0.358 0.99+ 0.590-0.744 0.597-0.750 0.192-0.336 0.99+
Table 3: F-score significance of selection methods compared with success measurwes. Project Selection and Evaluation Method Success CBA NPV/IRR Payback Budget constraint Probability assess 0.030(FN**) Requirement 0.073(FN*) Top management Subjective
Success Metric 1 - Time 0.906 0.338 -0.330 0.561 0.467 0.006(FN***) 0.722 -0.024(FN**)
Success Metric 2 - Cost 0.442 0.009(FN***) 0.866 0.000(FN***) 0.167
Success Success Metric Metric 3 - Use 4 - Impact -0.550 0.245 -0.479 0.845 0.257
0.323 0.169 0.860 0.630 0.522
0.169
0.102
0.518
0.273 -0.599
-0.580 0.391
0.880 0.075(FN*)
Success Metric 5 - Overall 0.555 0.225 -0.713 0.803
0.656 0.748
FOOTNOTES * Significant at 0.10 level. ** Significant at 0.05 level. *** Significant at 0.001 level. REFERENCES Bacon, C.J. (1992). The use of decision criteria in selecting information systems/technology investments. MIS Quarterly, 16(3), 335-353. Bajjaly, S. (1999). Managing emerging information systems in the public sector. Public Productivity & Management Review, 230), 40-47. Ballantine, J., & Stray, S. (1998). Financial appraisal and the IS/IT investment decision making process. Journal of Information Technology, 13, 3-14. Bannister, P., & Remenyi, D. (2000). Acts of faith: Instinct, value, and IT investment decisions. Journal of Information Technology, 75(3), 231-241. Belout, A., & Gauvreau, C. (2004). Factors influencing project success: The impact of human resource management. International Journal of Project. Management, 22, 111. Bozeman, B., & Bretschneider, S. (1986). Public management information systems. Public Administration Review, 46, 475-487. Bozeman, B., & Kingsley, G. (1998). Risk culture in public and private organizations. Public Administration Review, 58(2), 109-118. Bretschneider, S. (1990). Management information systems in public and private organizations: An empirical test. Public Administration Review, 50, 536-545. Brown, M. (2003). Technology diffusion and the "knowledge barrier" the dilemma of stake holder participation. Public Performance & Management Review, 26(4), 345359. Cabral-Cardoso, C, & Payne, R. L. (1996). Instrumental and supportive use of formal selection methods in R&D project selection. IEEE Transactions in Engineering Management, 43(4), 402-410. Capability Maturity Model Integration (CMMI). (1993). CMU/SEI-93-TR-24, Capability Maturity Model for Software, V1.1. Pittsburgh, PA: Carnegie Mellon University. Caudle, S., Gorr, W., & Newcomer, K. (1991). Key information systems management issues for the public sector. MIS Quarterly, 15(2). 171-188. Chapman, C. B., Ward, S. C, & Klein, J. H. (2006). An optimized multiple test framework for project selection in the public sector, with a nuclear waste disposal casebased example. International Journal of Project Management, 24(5), 373-384. Cheng, E.W. L., & Li, H. (2005). Analytic network process applied to project selection. Journal of Construction Engineering & Management., 131(4), 459-466. Dietz, R., & Renkema, T. (1995). Planning and justifying investments in information technology: A framework with case study illustrations. In A. Brown & D. Renkema (Eds.), Proceedings of the Second European Conference on Information Technology Investment Evaluation (pp. 28-41). Birmingham, UK: The Operational Research Society. Duarte, B. P. M., & Reis, A. (2006). Developing a projects evaluation system based on multiple attribute value theory. Computers & Operations Research, 33(5), 14881504. Evans, J. R., & Olson, D. L. (2003). Statistics, data analysis, and decision modeling (2nd ed.). Upper Saddle River, N.J: Prentice Hall. Farbey, B., Land, E, & Targett, D. (1992). Appraising investments in IT. Journal of information Technology, 7, 109-122. Hess, P. C., & Warren, C. S. (1993), Accounting principles. New York: Thomson. Finch, P. (2003, September). Applying the Slevin-Pinto project management profile to an information systems project. Project Management journal, 34(3), 32-39. Fitzgerald, G. (1998). Evaluating information systems projects: A multidimensional approach. JournaI of Information Technology, 13, 15-27. Flaig, J.J. (2005). Improving project selection using expected net present value analysis. Quality Engineering, 17(4), 535-538. Gabriel, S. A., Kumar, S., Ordóñez, J., & Nasserian, A. (2006). A multiobjective optimization model for project selection with probabilistic considerations. SocioEconomic. Planning Sciences, 40(4), 297-313. Ginzberg, M. J., & Zmud, R.W. (1988). Evolving criteria for information systems assessment. In N. Bjorn-Anderson & G. R. Davis (Eds.), Information systems assessment: Issues and challenges (pp. 41-52). Amsterdam: North Holland. Hinton, M., & Kaye, K. (1996). Investing in information technology: A lottery? Management Accounting, 74(10), 52. Hochstrasser, B. (1992). Justifying IT investments. Proceedings of the Advanced Information Systems Conference (London) (pp. 17-28). Jiang, J.J., & Klein, G. (1999). Information system project-selection criteria variations within strategic classes. IEEE Transactions on Engineering Management, 46(2), 171176. Jiang, J., Klein, G., & Balloun, J. (1996, December). Ranking of system implementation success factors, Project Management Journal, 27, 49-53. Keil, M. (1995). Pulling the plug: Software project management and the problem of project escalation. MIS Quarterly, 19(4), 421-447. Keil, M., Mann, J., & Rai, A. (2000). Why software projects escalate: An empirical analysis and test of four theoretical models. MIS Quarterly, 24(4), 631-664. Klein, G., & Beck, P. O. (1987). A decision aid for selecting among information system alternatives. MIS Quarterly, 11(2), 177-185. Kulak, O., Kahraman, C., Öztaysi, B., & Tanyaçs, M. (2005). Multi-attribute information technology project selection using fuzzy axiomatic design. Journal of Enterprise Information Management, 18(3), 275-288. Liesiö, J., Mild, P., & Salo, A. (2007). Preference programming for robust portfolio modeling and project selection. European Journal of Operational Research, 181(3),
1488-1505. Mohanty, R. P., Agarwal, R., Choudhury, A. K., & Tiwari, M. K. (2005). A fuzzy ANP-based approach to R&D project selection: A case study. International Journal of Production Research, 43(24), 5199-5216. Mullen, R., & Donnelly, J. T. (2006). Keeping it real -- Building an ROI model for an ambulatory EMR initiative that the physician practices espouse. Journal of Healthcare Information Management, 20(1), 42-52. Nutt, P. (1999). Public-private differences and the assessment of alternatives for decision making. Journal of Public Administration Theory and Research, 9, 305-350. Olson, D. (2004). Introduction to project management (2nd ed.). New York: Irwin/McGraw-Hill. Omitaomu, O. A., & Badiru, A. B. (2007). Fuzzy present value analysis model for evaluating information system projects. The Engineering Economist, 52, 157-178. Paisittanand, S., & Olson, D. L. (2006). A simulation study of IT outsourcing in the credit card business. European Journal of Operational Research, 175(2), 1248-1261. Patlon, N., & Shechet, A. (2007). Earned value management: Are expectations too high? CrossTalk, 20(1), 10-15. Pinto, J., & Slevin, D. (1989). Critical success factors in R&D projects. Research in Technology Management, 32(1), 31-35. Powell, P. (1992). Information technology evaluation: Is it different? Journal of the Operational Research Society, 43(1), 29-42. Scheil, G. P. (1986). Establishing the value of information systems. Interfaces, 16(3), 82-89. Sonde, T. (2004). 11 ways to leave your legacy systems behind -- FAR behind. Government Procurement, 120), 6-8. The Standish Group. (2004). Third Quarter Research Report. Retrieved January 22, 2008, from http://www.standishgroup.com/index.php Tam, K. Y. (1992). Capital budgeting in information systems development. Information and Management, 23, 345-357. Weill, P. (1990). Strategic investment in information technology: An empirical investigation. Information Age, 12(3), 143-147. Welsh, W. (2004). Governments eye slight spending increases. Washington Technology, 19, 6. Yu, A. G., Flett, P. D., & Bowers, J. A. (2005). Developing a value-centered proposal for assessing project success. International Journal of Project Management, 23(6), 428-436.
