International Journal of Forecasting 19 (2003) 5–25 www.elsevier.com / locate / ijforecast

Conducting a sales forecasting audit Mark A. Moon*, John T. Mentzer, Carlo D. Smith Department of Marketing, Logistics and Transportation, 301 Stokely Management Center, The University of Tennessee, Knoxville, TN 37996 -0530, USA

Abstract Continuous improvement in sales forecasting is a worthy goal for any organization. This paper describes a methodology for conducting a sales forecasting audit, the goal of which is to help a company understand the status of its sales forecasting processes and identify ways to improve those processes. The methodology described here has been developed over a 5-year period, involving multiple auditors, and has been implemented (to date) at 16 organizations. This methodology revolves around three distinct phases: the ‘as-is’ phase, in which the audit team seeks to understand fully a company’s current forecasting process; the ‘should-be’ phase, in which the audit team presents a vision of what world-class forecasting should look like at the audited company, and the ‘way-forward’ phase, in which the audit team presents a roadmap of how the company can change its forecasting processes to achieve world-class levels. Those companies that have responded positively to the audit process have experienced significant improvement in their forecasting performance. The paper concludes by presenting lessons from audits conducted to date, as well as implications for management practice and future research.  2002 International Institute of Forecasters. Published by Elsevier Science B.V. All rights reserved. Keywords: Forecasting practice; Audit; Forecasting management; Performance measurement; Forecasting systems; Supply chain

1. Introduction Forecasting has been consistently recognized as an important capability for business planning and management (Armstrong, 1987; Cox, 1987, 1989; Fildes & Hastings, 1994; Makridakis & Wheelwright, 1977; Mentzer & Gomes, 1994; Sanders & Manrodt, 1994; Wright, 1988). Regardless of industry, or whether the company is a manufacturer, wholesaler, retailer, or service provider, effective demand forecasting helps organizations identify market opportunities, enhance channel relationships, increase customer satisfaction, *Corresponding author. Tel.: 11-865-974-8062; fax: 11-865974-1932. E-mail address: [email protected] (M.A. Moon).

reduce inventory investment, eliminate product obsolescence, improve distribution operations, schedule more efficient production, and anticipate future financial and capital requirements (Galfond, Ronayne, & Winkler, 1996; McIntyre, Archabal, & Miller, 1993). To improve forecasting, companies increasingly take advantage of computer and information system technologies. Point of sale (POS) data collection is gathering near-real-time sales movement at a stock keeping unit by location (SKUL) level of detail (Smart, 1995). Electronic data interchange (EDI) is used to transmit detailed sales information throughout various marketing channels to product distributors and manufacturers (Mentzer & Kahn, 1997). Forecasting systems that were once housed on

0169-2070 / 02 / $ – see front matter  2002 International Institute of Forecasters. Published by Elsevier Science B.V. All rights reserved. PII: S0169-2070( 02 )00032-8

6

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

large central mainframes are now capable of running on desktop computers and in client server environments (Mentzer & Kahn, 1997; Mentzer & Schroeter, 1993, 1994; Mentzer & Kent, 1999). Legacy systems, which have historically been limited to a single forecasting technique for all products and services, are being replaced by systems which select from a series of alternative forecasting techniques or employ a combination of techniques to analyze demand and related information in an effort to improve forecasting accuracy (Mentzer & Kahn, 1997; Mentzer & Schroeter, 1993, 1994; Mentzer & Kent, 1999; Wright, 1988). Yet, despite these advances, forecasting sophistication and performance have improved little at even the more successful companies (Mentzer & Kahn, 1995). While forecasting research continues to pursue improvements in systems and techniques (Mentzer, 1988; Mentzer & Gomes, 1994; Smith, McIntyre, & Achabal, 1994), recent studies and reviews have identified gaps in our understanding of the relationships between systems and techniques used for forecasting, and the behavioral factors associated with the management of forecasting in organizations (Armstrong, 1987; Fildes & Hastings, 1994; Mentzer, Bienstock, & Kahn, 1999; Winklhofer, Diamantopoulos, & Witt, 1996). As Fildes and Hastings (1994) summarized, ‘most of that research and almost all of the text books have concentrated on only one aspect of the problem: how to develop appropriate forecasting methods’. To focus some future forecasting research less on methods and more on management practice, the purpose of this paper is to describe a methodology for conducting a sales forecasting audit that has been tested in 16 companies to: (1) understand the current status of their forecasting management practices (the ‘as-is’ state); (2) visualize the goals they should be striving to reach in the various dimensions of forecasting management (the ‘should-be’ state), and (3) develop a roadmap for achieving their goals (the ‘way-forward’ process). (At the time of acceptance of this manuscript (August 2001), the following companies had participated in the sales forecasting audit research: Allied Signal, Avery Denison, ConAgra, Corning, DuPont, Eastman Chemical, Ethicon, Exxon, Hershey Foods USA, Lucent Technologies, Michelin North America, Motorola,

Pharmavite, Smith and Nephew, Union Pacific Railroad, and Williamson–Dickie.) Section 2 of this paper discusses the relevant research in forecasting management. In Section 3, a standard against which companies can compare their forecasting management practices is discussed. This standard provides the basis of the sales forecasting audit process, which is then discussed in detail. Sections 4 and 5 discuss research and managerial implications of this forecasting management research.

2. Forecasting management research In a 1977 review of forecasting literature, Makridakis and Wheelwright presented three areas where forecasting posed issues and challenges for management: (1) the range of alternative forecasting methods; (2) the selection of forecasting methods in practice, and (3) organizational and behavioral factors affecting the forecasting environment. While they recognized the need to continue research to develop alternative forecasting techniques, they emphasized a need to focus more on the ‘problemoriented or application side of forecasting’ (1977, p. 36). Such a focus, they indicated, would ‘lead to a stronger base on which management can effectively use forecasting knowledge’ (1977, p. 36). A considerable amount of subsequent forecasting research has focused on the development and selection of forecasting methods. A review of research published in a variety of business journals—including those in forecasting, marketing, logistics, operations management, and management science—indicates a relatively limited number of efforts to answer Makridakis and Wheelwright’s (1977) call to incorporate more applications oriented and behavioral research. Since the purpose of this paper is to focus on forecasting management, the following discussion will address this forecasting management research. The few articles that discuss the area of forecasting system implementation and administration typically propose an implementation process and present a case study to illustrate the process and describe results. Examples of such studies include Armstrong (1987), Schultz (1984), Closs, Oaks, and Wisdo (1989), Fildes and Hastings (1994), Mentzer (1999),

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

Mentzer and Kent (1999), and Mentzer and Schroeter (1994). These articles suggest that forecasting implementation entails more than the application of more accurate forecasting techniques and that effective forecasting management, as with effective system implementation, may lead to improved operating and business performance. Three exemplary articles from this stream of research bear elaboration. The first (Armstrong, 1987) presented an idealized forecasting case study and, from this case, derived 16 ‘pitfalls in forecasting’. From these pitfalls, Armstrong developed a forecasting audit checklist for companies to review their own management processes. This audit included the steps: (1) assess the methods without the forecasts, (2) assess the assumptions and data used in the forecast, (3) assess the uncertainty of the forecast, and (4) assess the costs of the forecast. Although not based upon an actual company, this paper did establish the importance of auditing company forecasting processes instead of just methods. The second paper (Fildes & Hastings, 1994) drew upon the organizational behavior and diffusion literature to develop a theoretical model of a marketing forecasting system. From this model, a series of theoretical statements were derived in the areas of (1) the forecaster and the decision maker, (2) information flows, and (3) technical characteristics of the forecast. The authors then tested their theoretical statements in a case study of a company with 10 separate divisions and concluded that: • techniques and forecasting software should meet the needs of the forecaster; • with limited time and technical expertise, implementation will also critically depend on the database system and the supporting organization design; • without accountability for forecast improvements and adequate time and resources, little will happen; and • forecast improvement depends on organizational design. The importance of Fildes and Hastings (1994) research to this paper is the establishment of a theoretical basis from which to develop a model of

7

forecasting management, and the use of case companies to test the model. In the third article, Schultz (1984) discussed managerial issues surrounding the successful implementation of new forecasting models. He outlined 12 ‘implementation profile factors’ which he summarized from his own and other forecasting management research. These factors include top management support, the relationship between forecast users and model designers, goal congruence, implementation strategy and resources, and cost–benefit justification. The importance of the Schultz (1984) paper is its explicit recognition that even the best forecasting models will not affect overall corporate performance without attention to managing organizational change processes. While these studies have provided valuable information and insight into forecasting management, they have not offered a generalizable framework that may be used as a prescriptive tool to improve forecasting management. As Armstrong (1988) pointed out, a key area where researchers have yet to provide assistance to practitioners is in implementation. In other words, how can practising forecasters take the knowledge that has been developed about forecasting, and put it into action to improve forecasting in their particular organization? It is this question that the remainder of this paper addresses.

3. Conducting a forecasting audit The purpose of the research described here was the development of a process by which companies could review their sales forecasting processes and practices to improve sales forecasting performance. An important aspect of the process is its applicability to organizations of different sizes, structures, and industries. The findings presented in this section are based on in-depth research conducted at 16 companies.

3.1. The role of auditing An audit has been defined as ‘a formal evaluation of performance to predetermined standards and the use of that evaluation to induce improved performance’ (Arter, 1989). While auditing is typically

8

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

thought of in relation to preparation of financial statements, other areas of business use auditing as a way to arrive at an unbiased assessment of current performance, as well as identify areas of needed improvement. Examples include marketing audits (Tybout & Hauser, 1981), sales management audits (Churchill, Ford, & Walker, 1993), human resource management audits (Hussey, 1995), and quality audits (Dew, 1994). One of the characteristics of a successful audit emphasized by each of these authors is the unbiasedness of the auditors themselves. Each author encourages engagement of credible experts from outside the organization to both collect and analyze the data. There are three reasons for this. Firstly, outside experts have knowledge of accepted standards against which current management practice can be compared. As Fildes and Ranyard (1997) point out, external consultants ‘have a wide knowledge of business practices across a range of organizations, as well as relevant knowledge of competitor operations. They may also have specialist knowledge and skills which few in-house groups can sustain’ (p. 339). Secondly, individuals from outside the organization have no incentive to overlook areas that might prove sensitive or embarrassing to current management. Thirdly, data collection, particularly when such data collection involves interviews with individuals currently involved in a management process, can be more successful when those collecting the data are from outside the organization. Individuals tend to be more willing to share their true experiences with unbiased, outside experts, particularly with assurances of confidentiality. Or, as Fildes and Ranyard (1997) point out, for an internal project or audit team, ‘confidentiality may be perceived as more problematic compared to an external consultancy’ (p. 339). As mentioned above, an audit can be seen as a ‘formal evaluation of performance to predetermined standards’. Thus, before a description of the sales forecasting audit process can begin, we turn to a discussion of relevant ‘predetermined standards’ for sales forecasting management.

3.2. Best practices in sales forecasting management Before practising forecasting managers can know

whether or not their organizations’ forecasting processes are good or bad, it is useful to have a standard against which those processes can be compared. Armstrong (1987) provided such a standard with his 16-point forecasting audit checklist. This checklist was divided into the dimensions of ‘forecasting methods’, ‘assumptions and data’, ‘uncertainty’, and ‘costs and benefits’ (see Table 1). Fildes and Hastings (1994) utilized organizational behavior and diffusion theory to present a normative model of the Marketing Forecasting System. Based upon this model, the authors suggested evaluating the forecaster and the decision maker, information flows, and the technical characteristics of the forecast (see Table 1). Mentzer et al. (1999) based their four-dimensional framework on the work of Armstrong (1987), Fildes and Hastings (1994), and Schultz (1984), and the findings from a 15-year, three-phase research program in forecasting management. Phases one (Mentzer and Cox, 1984a) and two (Mentzer and Kahn, 1995) were based upon two large survey-based quantitative analyses of sales forecasting practices. Phase three involved a benchmark study of the forecasting practices at 20 leading companies representing a variety of industries. Based on in-depth analyses of company processes and documents, as well as interviews with both users and developers of forecasts, Mentzer et al. (1999) proposed that, in order to adequately understand the overall management of the forecasting process in a company, that process must be investigated along the following four dimensions. 1. Functional integration, concerned with the role of collaboration, communication, and coordination of forecasting management with the other business functional areas of marketing, sales, finance, production, and logistics. 2. Approach, concerned with which products and services are forecast, the forecasting techniques used, and the relationship between forecasting and planning. 3. Systems, addressing the evaluation and selection of hardware and software combinations to support the sales forecasting function as well as the integration of forecasting systems with other planning and management systems in the organization.

Table 1 Forecasting management frameworks Fildes and Hastings (1994)

Mentzer et al. (1999)

Forecasting methods 1. Forecasting independent of top management? 2. Forecast used objective methods? 3. Structured techniques used to obtain judgments? 4. Least expensive experts used? 5. More than one method used to obtain forecasts? 6. Users understand the forecasting methods? 7. Forecasts free of judgmental revisions? 8. Separate documents prepared for plans and forecasts?

The forecaster and the decision maker 1. Forecaster’s managerial style 2. Forecaster’s training 3. Link between formal forecast and user’s decision

Functional integration 1. Degree of communication, coordination, and collaboration between forecasting group and other functional areas 2. Organizational location of the forecasting group 3. Existence and form of consensus forecasting meetings 4. Recognition of forecasting needs of various functional areas 5. Accountability/performance rewards for personnel involved in developing the forecasts

Assumptions and data 9. Ample budget for analysis and presentation of data? 10. Central data bank exists? 11. Least expensive macroeconomic forecasts used? Uncertainty 12. 13. 14. 15.

Upper and lower bounds provided? Quantitative analysis of previous accuracy provided? Forecasts prepared for alternative futures? Arguments listed against each forecast?

Information flows 4. Information flow from the environment 5. Intraorganizational information flows and loss of information 6. IT support for information flows Technical characteristics of the forecast 7. Accuracy and bias 8. Responsiveness and speed 9. Uncertainty estimation

Approach 6. 7. 8. 9. 10. 11.

Costs and benefits 16. Amount spent on forecasting reasonable?

Systems 12. 13.

Relationship between forecasts and plans Orientation of the forecasting approach (top-down or bottom-up) What is forecast in the supply chain? Forecasting segmentation of products by importance Use of quantitative and qualitative forecasting techniques Training in technique usage

Intracompany and supply chain electronic links Information availability (reports and performance metrics) 14. Degree of systems knowledge in the organization Performance measurement 15. Measurement and use of accuracy 16. Recognition of the impact of external factors on accuracy 17. Measurement and use of other performance measures (costs and customer service)

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

Armstrong (1987)

9

10

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

4. Performance measurement, considering the metrics used to measure sales forecasting effectiveness and its impact on business operations. In addition to identifying these four dimensions of forecasting management, Mentzer et al. (1999) articulated four stages of effectiveness within each dimension (see Figs. 1–4). Their article provided a detailed description of the characteristics that can be found at each of the four stages of effectiveness within each of the four dimensions. From the Table 1 comparison of the Armstrong (1987), Fildes and Hastings (1994), and Mentzer et al. (1999)

frameworks, we can make several observations. Armstrong (1987) concentrated more on the methods aspect of forecasting and, thus, provided more evaluative detail on forecasting methods than the other two frameworks. The Armstrong (1987) framework can be largely subsumed under the Fildes and Hastings (1994) category of ‘technical characteristics of the forecast’, and the Mentzer et al. (1999) category of ‘approach’. The Fildes and Hastings (1994) framework provides a more comprehensive framework than Armstrong (1987), with many of the same evaluative criteria as Mentzer et al. (1999): the criteria under ‘the forecaster and the decision maker’

Fig. 1. Forecasting benchmark stages: functional integration.

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

11

Fig. 2. Forecasting benchmark stages: approach.

correspond to similar criteria under ‘functional integration’; ‘information flows’ correspond to both ‘functional integration’ and ‘systems’; and ‘technical characteristics of the forecast’ correspond to much in the ‘approach’ category.

Further analysis of these three studies reveals certain key ‘themes’ in sales forecasting management (Table 2). Table 2 takes the numbers of the key elements of each study articulated in Table 1 and categorizes them under 24 exemplars of key themes.

12

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

Fig. 3. Forecasting benchmark stages: systems.

These key sales forecasting management themes, which run across all three studies, pay attention to: the organization, information, technical issues, the forecaster and users, and costs and benefits of forecasting effectiveness. Perhaps due to the benefit of the existence of the Armstrong (1987) and Fildes and Hastings (1994) frameworks and the fact that the Mentzer et al. (1999) framework was based on the evaluation of a broad set of companies, the Mentzer et al. (1999) dimensions and stages appear to provide the most detailed and comprehensive comparison standard. However, several exemplars were addressed by Armstrong (assumptions explicit, reasons for forecast uncertainty explored, users’ knowledge, and resources available) and / or Fildes and Hastings

(reasons for forecast uncertainty explored, variables forecast and lead time, and forecaster’s style) that are not specifically addressed by Mentzer et al. (1999). Since it was the purpose of this auditing research not to refine any particular framework but rather to select one that could be used as a standard against which company processes could be compared, the combination of these 24 exemplars into one more comprehensive framework is left to future research. Therefore, since the Mentzer et al. (1999) framework was the most comprehensive, it was adopted for this study. Since the forecasting audit process that we describe below adopted this framework, the details of the stages in each of the Mentzer et al. dimensions are reproduced here in Figs. 1–4. Rather than directly reproduce the figures from the Mentzer

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

13

Fig. 4. Forecasting benchmark stages: performance measurement.

et al. (1999) article, we have substituted actual figures from four different forecasting audits as exemplars of audit findings. Each of the four figures is taken from a different audit. The bullet points in bold, italic type represent the audit team’s assessment of that particular company’s status on that particular dimension of forecasting management. These bolded, italicized bullet points will be explained in more detail later in the paper.

3.3. The audit process—data collection The audit process developed in this research was used to assess practices and recommend actions to improve forecasting management performance at 16 large and diverse organizations. Table 3 is provided to show the diverse nature of the organizations that

have participated to date in the audit research. While these organizations are diverse in terms of products and services they offer, one important commonality they all shared was a realization that their forecasting practices were in need of improvement. Each of the 16 organizations agreed to participate because they felt the audit could help them identify and rectify fundamental problems with their forecasting practices. As a result, while it is impossible easily to characterize the ‘typical’ company that has participated in the audit research to date, one characterization is that, on each of the four dimensions of forecasting management discussed above, the ‘typical’ company is in stages one or two on all four dimensions. It is in this way that this research diverges from Mentzer et al. (1999). Where Mentzer et al. developed their dimensions while investigating

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

14

Table 2 Themes across sales forecasting management frameworks a Themes

Exemplars

MBK

F&H

Armstrong

Organization

Communications Coordination Location Interaction Link to planning and decisions

1 1 2 3 4, 5, 6

3

1, 8

From environmental Within supply chain Intra-organization Availability Information systems

12 8 12 13 14

Top-down / bottom-up Techniques Product segmentation Assumptions explicit Measurement of accuracy Impact of external factors Reasons for forecast uncertainty explored Variables forecast and lead time

7 10 9

Information

Technical issues

The forecaster and users

Costs and benefits a

Training Incorporation of judgement Forecaster’s style Users’ knowledge Resources available Value (perceived and actual) to organization

15 16

4 4 5 6

10

2, 3, 4, 5, 7, 11

7 9

15 13 14 12

8

11 10

2 1 6

17

3

9 16

The authors wish to thank Professor Robert Fildes for the original idea and guidance to develop this table.

companies possessing a wide range of forecasting proficiency on each of the four dimensions, the purpose of this research was to focus on companies with a recognizable deficiency in one or more of the dimensions. Our purpose was to see if these dimensions could serve as a diagnostic tool to help companies improve their forecasting performance. The process used to conduct a forecasting audit is graphically depicted in Fig. 5. This process begins with identification of the liaison person within the company. Because the companies that agree to participate in the audit research are companies that recognize their own deficiencies, this individual is the person who has been charged with initiating a forecasting re-engineering effort. The liaison helps

coordinate the details of the audit, as well as choose the individuals to be interviewed in the data collection phase. The next step is an analysis of all relevant documentation. Prior to on-site data collection, it is useful for the audit team to become familiar with the forecasting process as it is currently understood by those responsible for forecasting. Thus, any written documentation that describes information flows, reports that are available to forecasters or users, organization charts, hardware and software systems descriptions and documentation, historical accuracy figures and reports, and uses of the forecasts are analyzed by the audit team. Of the 16 companies that have been audited, there

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

15

Table 3 Characteristics of audited companies Company

Consumer?

Allied Signal Avery Denison ConAgra Corning DuPont Eastman Chemical Ethicon Exxon Hershey USA Lucent Technologies Michelin North America a Motorola b Pharmavite Smith & Nephew Union Pacific RR Williamson–Dickie

X X X

Totals

8 (50%)

a b

Business to business?

Primarily direct sales?

X X X X X

X X X X X

X

X X X

X X X

X X

X

X X

X X X X

Primarily sales through distributors?

X

X

X X X X X

10 (63%)

6 (38%)

12 (75%)

Michelin’s business is divided between OEM and replacement tire. Motorola’s business involved wireless and pager business to consumers and business customers.

has been considerable variance in the quality and completeness of the documentation provided before the on-site visit. At one extreme was a 5-inch-thick binder containing an extremely comprehensive description of the current system—including detailed descriptions of what happens during each month and throughout the fiscal year of the forecasting process—along with hundreds of pages of reports that can be generated on demand by the forecasting system. At the other extreme was a photocopy of the software manual for the forecasting system that was installed, but never used, and no documentation of processes. In addition to receiving information from the company being audited, the audit team provides a detailed, eight-page interview protocol to the audit sponsor. This protocol presents detailed questions on the four dimensions suggested by Mentzer et al. (1999), i.e. how forecasts are prepared, the systems that are used to support forecasting, the techniques that are employed, what (if any) approaches are taken to measuring forecasting performance, and how the forecasts are used. (Copies of the detailed protocol are available on the web site, http: / / bus.ut-

k.edu / forecasting.) The sponsor typically provides copies of this protocol to the interview participants to help them prepare for their interviews. The protocol is designed to help those who are to be interviewed understand the type of information that the audit team is trying to collect. It is never the case that any single individual is able to answer all the questions posed in the protocol. Rather, the protocol is meant to guide the audit team through the entire data collection process, and provide those interviewed with guidance to reflect on issues in preparing for the interviews. In other words, by the completion of the on-site visit, the audit team’s objective is to have answers to all the various items in the protocol from the combined interview responses. Although it is the responsibility of the sponsor to select the individuals to be interviewed, the audit team should communicate the critical importance to the success of the audit that an appropriate range of individuals be included on the interview list. The participant list needs to be both ‘broad’ and ‘deep’. Broad means adequate representation from all the different functions in the company that are involved in developing or using the sales forecasts. It is

16

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

Fig. 5. Forecast audit data collection process.

important that at least three different groups are represented: those who provide input to the sales forecast (e.g. the sales and marketing organization), those who actually prepare the sales forecast (i.e. the sales forecasting group), and those who are customers of the sales forecast (e.g. purchasing, production planning, logistics planning, and finance). The participant list also needs to be ‘deep’ in the sense that it includes various levels of the organizational hierarchy. If only senior level managers are interviewed, there is a danger they will have inadequate understanding of the detailed issues and problems faced by the people who actually do the

work. At the same time, if only ‘workers’ are included, there is the danger interviews will become little more than complaint sessions, and the audit team may be left with an insufficient understanding of the strategic issues surrounding the sales forecasting process. Therefore, it is up to the audit sponsor carefully to select and schedule the right individuals to participate in the interviews. Following this preparation is the on-site visit, conducted by a team of four auditors. The team of four auditors splits into two sub-teams so that two auditors are present for each interview, providing the ability to assess inter-rater reliability, a crucial

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

component of validity and reliability in interviewbased research (Armstrong, Gosling, & Weinman, 1997; Griggs, 1987; Hughes & Garrett, 1990; Kurasaki, 2000). Each interview is audiotaped so any differences in interpretation can be resolved at the time of data analysis. Interviews typically include one subject and two auditors, although occasionally two or more subjects are present during a single interview. The interview typically begins with one member of the audit team briefly describing the audit’s purpose, giving assurances of confidentiality, and obtaining permission to audiotape the conversation. The auditors then ask the subject to describe his or her role in the forecasting process. Probes include questions such as ‘what do you do with that information?’, ‘where do you obtain that information?’, and ‘is the information you obtain from that source credible and complete?’ While a considerable portion of the interview is spent gaining an understanding of the subject’s formal role and responsibility with regard to the forecasts, the auditors also probe to obtain insights as to the subject’s satisfaction with the sales forecasts and the sales forecasting process. Focus is placed on understanding any frustrations or problems the subject has with the forecasting process and his or her role in that process. Each interview ends with a ‘wish-list’ question, where one of the auditors asks the subject to describe what he or she would do to make the forecasting process more effective. Interviews typically last 45 min to an hour. At the 16 companies audited to date, the number of interviews has ranged from 22 to 64, with an average of 32 per company. Following completion of the interviews, the auditors combine all interview notes, then distribute those notes to all audit team members. With data collection completed, data analysis begins.

3.4. The ‘ as-is’ status The purpose of the sales forecasting audit is to articulate for the audited company its ‘as-is’ status, a vision of its ‘should-be’ position, and a description of the ‘way-forward’ process that will help the company improve its sales forecasting practices. The first step is for the research team to understand fully

17

the current status of the company’s forecasting practices (the ‘as-is’). As each member of the audit team analyzes the data, two primary objectives are foremost. Firstly, the company’s position on each of the four dimensions of sales forecasting management—functional integration, approach, systems, and performance measurement—is assessed. Each member of the audit team identifies characteristics of the company being audited, and compares those characteristics to the bullet points in Figs. 1–4. Initially, each research team member does this work independently, and then all members meet to compare the results of their analyses. When disagreements arise, the analysts talk through their reasons for highlighting a particular bullet point on one of the four dimensions, citing evidence from the documentation and / or the interview notes. If necessary, the audiotapes of the interviews are replayed for clarification, and consensus is reached. The results of this process are a version of Figs. 1–4, with various bullet points, or portions of bullet points, highlighted to identify characteristics of the audited company on each of the four dimensions of forecasting management. As mentioned previously, Figs. 1–4 give actual examples from audits that have been conducted to date. Examination of these figures shows how different bullet points have been highlighted, identifying the ‘as-is’ state of that particular company on each of the four dimensions. It is important to note that during the analysis it is common to identify characteristics consistent with multiple stages of sophistication along a forecasting dimension. For example, examination of Fig. 1 reveals that on the ‘functional integration’ dimension, this particular company exhibited one world class, stage four characteristic—existence of forecasting as a separate functional area—while also exhibiting several stage one and stage two characteristics. The insight revealed here is that while well positioned organizationally to achieve a high level of functional integration, lack of a forecasting champion (Mentzer, Moon, Kent, & Smith, 1997) has prevented this company from taking advantage of this organizational strength. Similarly, Fig. 2 shows a company that, on the ‘approach’ dimension, has four stage one characteristics, three stage two characteristics, and one stage three characteristic. In other words, some areas of the

18

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

‘approach’ dimension have great opportunities for improvement, other areas have lesser opportunities for improvement, and still other areas on the ‘approach’ dimension are managed quite well. As will be discussed in the next section (the ‘should-be’ state), this articulation of characteristics at different stages of sophistication helps a company identify and prioritize areas where improvement can take place. The audit process is not concerned with classifying the company solely within a single discrete stage of sophistication as much as understanding the relationships among the stage characteristics and the implications for improving performance. The second objective of analyzing the interview notes is to identify ‘strategic themes’ that emerge from the data. These strategic themes are issues that cut across multiple dimensions of forecasting management, and which are so pervasive or which cause such wide-ranging problems that they demand special attention and discussion. Following is a discussion of those strategic themes that emerged at a majority of the companies. • Limited performance measurement and lack of performance evaluation (12 of 16 companies)— while this characteristic is discussed at length in the dimension on ‘performance measurement’, it is such a pervasive characteristic that at most companies it has warranted special discussion as a strategic theme. One clear lesson learned as a result of this research is companies do not seem to measure adequately forecasting performance, tie that forecasting performance to the evaluation of individuals, and then reward individuals for excellence in forecasting. • Blurred distinction between forecasts, plans, and goals (11 of 16 companies)—this is a situation where a company does not recognize that forecasts are a projection into the future of expected demand, given a stated set of environmental conditions, while plans are managerial actions proposed by the organization to capture and supply as much of the forecasted demand as possible (Mentzer & Bienstock, 1998). Evidence of this theme can be found in these actual statements from audits: ‘we forecast up to plan’, or ‘it would be suicide for me to forecast anything different than the plan’. Such statements indicate

these organizations are creating forecasts based upon plans or sales targets, rather than their best judgments about future customer demand. • Limited commitment to sales forecasting (10 of 16 companies)—this theme was manifested by a number of different situations at audited companies, including: insufficient commitment of resources to training, documentation, systems support, or reward and recognition programs; relegating the forecasting function to relatively low levels in the organizational hierarchy; unwillingness to designate a forecasting champion, and lack of accountability throughout the organization for forecast accuracy. At one company, this theme was manifested by the failure to fill an open director-level forecasting position for over a year. Because of this lack of leadership, the company’s forecasting improvement efforts were unfocused and unsupported by other constituent organizations in the company. • Islands of analysis (nine of 16 companies)—this is the situation where a company has non-standard, non-interfacing systems or procedures for performing similar tasks, or forecasting systems that fail to connect with other enterprise systems like production planning or finance. These ‘islands’ can range from each forecasting analyst having his or her own ‘home-made’ spreadsheet with unique characteristics and assumptions, to separate forecasting systems installed and operating in different departments of the company, to the manual transfer of data either into or out of a forecasting system. An extreme example of this phenomenon occurred at one audited company, where three separate forecasting systems had been installed over time: a mainframe-based legacy system, which was the ‘official’ forecasting system, an AS / 400-based system installed by production planning, and a PC-based system installed by logistics. These latter two were described as ‘black market’ forecasting systems, and were installed because the forecast user organizations did not trust the integrity of the ‘official’ forecast, and so created their own forecasting systems. From these two sources (dimensions and strategic themes), the portrayal of the audited company’s ‘asis’ state of forecasting management practices is

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

complete. The second stage of the audit process is the description of the ‘should-be’ state of sales forecasting management practices for the company.

3.5. The ‘ should-be’ state While understanding the current status of sales forecasting management (the ‘as-is’ state) is important, managers cannot take steps toward excellence without guidance on the directions to take. For this reason, it is important to provide a clear picture of the ‘should-be’ state of forecasting management in the company. This ‘should-be’ picture can be found in Figs. 1–4 in two different ways. The first is to provide broad targets at which managers can aim. Examination of stage four in each of the four dimensions describes the most advanced level of forecasting excellence uncovered in the sales forecasting benchmark research. Therefore, a ‘best practices’ company would operate at stage four on all four dimensions. Stage four characteristics provide managers with a long-term target toward which they can strive. However, since few companies have achieved a level of excellence near stage four, it is important that intermediate targets be set to move toward stage four. If, for example, a company’s ‘as-is’ status is primarily in stage one, then stage two characteristics can be seen as intermediate targets that will improve forecasting effectiveness, and begin the company on the path to the excellence found in stage four. The second way that Figs. 1–4 provide the ‘should-be’ picture is in a more detailed, tactical sense. Careful examination of Figs. 1–4 reveals that for many of the bullet points found in each dimension, there is a natural progression from stage one (low level of forecasting sophistication) to stage four (high level of forecasting excellence). In Fig. 1, for example, which describes the dimension of ‘functional integration’, the first bullet under stage one describes a state where major disconnects exist between marketing, finance, sales, production, logistics, and forecasting. For one company that found its current ‘as-is’ state described by this bullet, the immediate ‘should-be’ target was found in the first bullet of stage two (coordination (formal meetings) between marketing, finance, sales, production, logis-

19

tics, and forecasting). In this more detailed way, a company can see where it should be going in each of the four dimensions of forecasting. Mentzer et al. (1999) asserted that stage four targets in each of the dimensions provide a benchmark to which the company should ultimately strive, while other, lower dimensions provide intermediate targets. In addition to their benchmark, the benefits of achieving a stage four level of sophistication along each dimension receives support from research in forecasting technique development and application, systems design and implementation, and management. The ‘functional integration’ characteristics of communication, coordination, and collaboration reflect the benefits associated with team-based forecasting (Kahn & Mentzer, 1994), and are identified as one of seven keys to better forecasting (Moon, Mentzer, Smith, & Garver, 1998). Maintaining a separate forecasting function is suggested as a means to reduce bias and support forecast processes (Fildes & Hastings, 1994). Providing forecasts in formats that match the requirements of user functions helps improve understanding and input (Mentzer & Bienstock, 1998; Marien, 1999; Fliedner, 2001; Mentzer & Schroeter, 1994). Forecast development is also viewed as a consensus building process, acknowledging the relationship between unconstrained market forecasts and the constraints associated with operating capabilities or requirements (Fildes & Hastings, 1994; Waddell & Sohal, 1994). Schultz (1992) emphasizes the need for multidimensional performance metrics noting that, ‘we must go beyond measures of accuracy and look to objective performance measures such as sales, costs, and profits’ (p. 410). Reviewing characteristics associated with a stage four level of ‘approach’, top down / bottom up forecast development has been identified as a means to improve forecast performance over either approach separately (Kahn, 1998; Fliedner, 2001). The need for forecast reconciliation between sales and operations (Fildes & Hastings, 1994; Waddell & Sohal, 1994; Nelson, 1987), and an understanding of the impact of sales force gaming (Galfond et al., 1996), whether internally or from customers, are also proposed to impact forecast accuracy. Forecast education that goes beyond technique development (Mentzer & Cox, 1984b) and top management support

20

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

(Miller, 1985; Schultz, 1984) are also recognized as key elements of forecast success. Characteristics associated with a stage four level of sophistication along the ‘systems’ dimension reflect findings from case studies involving system development and implementation (Mentzer & Schroeter, 1993, 1994; Mentzer & Kent, 1999). Mentzer and Kent (1999) outlined the development and implementation of new processes and systems that helped The Longaberger Company establish a ‘system-centric’ approach to forecast development. Their discussion addressed the need for companies considering forecasting systems to ‘make the tool fit the problem’. Kahn and Mentzer (1996) discussed the benefits of incorporating EDI as a means to integrate supply chain demand information to improve data availability and accuracy, and subsequently reduce inventory costs. Their propositions are supported by studies evaluating the impact of forecasting information availability on variability in the supply chain (Chen, Drezner, Ryan, & Simchi-Levi, 1999). Chen et al. (1999) quantified the improvement in supply chain forecasting and inventory performance resulting from a shift in information availability from a decentralized to centralized model. Performance measurement characteristics were supported by Fildes and Hastings (1994), who recognized that environmental factors influence forecasting practices and performance. Calling on managers to make forecasting important and to ‘measure, measure, measure’, Moon et al. (1998) emphasized the need to implement measures of forecasting performance based on accuracy and its impact on operating performance. While an understanding of the ‘should-be’ state is critically important to the continuous improvement process, it is not very useful without an understanding of how to get to that ‘should-be’ state. For that reason, we now turn to the third purpose of the forecast audit: the description of the ‘way-forward’.

3.6. The ‘ way-forward’ The audit process provides the audited company with a ‘way-forward’ roadmap through a series of concrete recommendations. While the recommendations are unique for each company, based on the

current status of their forecasting practices, these recommendations usually fall into four categories. Two of these categories, systems and performance measurement, directly match the dimensions previously discussed. The other two categories, process and training, are designed to help companies in both functional integration and approach. Process recommendations refer to the way forecasts are created and used. One company, for example, was at stage one on the dimension of ‘functional integration’, so process recommendations included instituting a consensus forecasting process, where different people from different parts of the company work together in a forum characterized by open information-sharing to create a consensus forecast. On the other hand, another company’s forecasting practices were at stage one on the ‘approach’ dimension, and statistical tools were not used effectively to uncover patterns in historical demand data. Thus, the process recommendations included implementation of a process where baseline forecasts are generated statistically, then distributed to knowledgeable experts, such as sales or marketing people, for adjustment. Training recommendations refer to specific situations where company personnel who are involved in forecasting have inadequate skills or knowledge to perform their forecasting tasks effectively. For example, salespeople are usually in a position to provide forecasting intelligence, but in only one of the 16 companies in our database did salespeople have any training on why forecasting is important (functional integration), or how to make qualitative adjustments to baseline forecasts (approach). Thus, in almost all companies that constitute the audit database, such training programs targeted at problems in ‘functional integration’ and ‘approach’ were recommended. Similarly and surprisingly, in 13 of the audited companies the people in the forecasting group had received no training on how time series and regression analysis can be used to create baseline forecasts, so training programs targeted at this deficiency in the ‘approach’ dimension were recommended. System recommendations refer to the way computer and communication systems can be enhanced to develop and communicate forecasting information more effectively. For example, the ‘as-is’ status of one company showed forecasting systems were not

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

closely integrated with other corporate systems, in this case finance and MRP systems, resulting in manual transfers of data. Therefore, the system recommendations included creating electronic linkages that allow data transfers between systems. Also, in several companies where ‘island of analysis’ existed, characterized by multiple processes or systems performing similar tasks, system recommendations included specific procedures designed to eliminate such islands and standardize on a single set of forecasting processes. Finally, performance measurement recommendations refer to specific metrics that should be put into place to measure forecasting performance adequately. For example, in six companies the salespeople were asked to make adjustments to baseline forecasts, but the accuracy of those adjustments was not measured and communicated back to the salespeople. This resulted in a recommendation that such a measurement and feedback system be implemented. Similarly, in the 13 companies that had implemented some performance metrics, accuracy was the only metric used. Thus, other metrics designed to assess the impact of forecasting accuracy on overall supply chain costs and customer service were recommended. While recommendations have been included in each of the 16 audits conducted to date, there has been considerable variance observed in management’s responses to these recommendations. Management reaction has typically fallen into one of three categories, which we can characterize as either ‘address the problems’, ‘assign the blame’, or ‘why should I care?’. Companies that fall under the ‘address the problems’ category tend to have an organizational culture oriented toward solutions, regardless of which department is ‘to blame’. Companies in this category also tend to recognize that responsibility for complex organizational problems are usually shared across functions, and thus look for cross-functional solutions. Companies in the ‘assign the blame’ category tend to have an organizational culture oriented toward identifying the source of organizational problems, and when that source is identified, that department becomes responsible for solutions. Since forecasting problems tend to be cross-functional, it is usually impossible to identify a single source of forecasting problems. Thus, com-

21

panies in this category tend to bog down in assigning blame rather than in pursuing solutions. Companies in the ‘why should I care?’ category tend to view forecasting as an unimportant activity. In these companies, there is no understanding at the senior management level of how forecasting improvements can dramatically enhance the company’s key performance indicators. One company in particular provides an excellent example of the ‘address the problems’ response. This organization, which was in stage one on all four dimensions of forecasting management, responded by assigning a cross-functional project team first to perform a detailed review of the recommendations, then propose a prioritization scheme, followed by an action plan. Three months after the conclusion of the audit, this project team met with the audit team to review their proposed priorities and action plan, and this was followed by a 2-year-long effort to reengineer their approach to sales forecasting. As a result of that re-engineering effort, this company now approaches stage four on all four dimensions of forecasting management, and their supply chain costs have been reduced dramatically. In fact, the company estimates their entire implementation effort cost less than $1 million, but the savings in raw material purchasing costs alone (buying more on long-term contracts based on accurate forecasts rather than buying at the last minute on the spot market) during 1998 were in excess of $7 million. Thus, the return on investment from implementation has been considerable. It is important to note that this $7 million saving was seen by upper management as an accomplishment by all departments involved in the forecasting effort, not just purchasing. Another company provides an example of the ‘assign the blame’ response. At this organization, which was primarily at stages two and three on two of the dimensions and stage one on the other two dimensions, the auditors’ final presentation to senior management degenerated into a heated discussion between executives over which department was at fault for the problems identified by the audit team. While these executives agreed that problems existed, none were willing to acknowledge that performance improvements were needed in their individual departments. As a result, no consensus could be reached on how to effect change, and no re-engineer-

22

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

ing effort was carried out. Over the next 18 months, this company experienced considerable disruption and customer service problems due to an inability to forecast demand hitting their distribution centers. Finally, at an audited company that exhibited the ‘why should I care?’ response, during the question and answer portion of the final presentation to senior management, the executive vice president of marketing rose from his chair and stated that all the company really needed was a new ‘killer product’, and with such a new product, all this attention to forecasting improvement would not be necessary. At another company, the CEO spent the time during the final presentation to management doing other work and gazing absently around the conference room. At a third company, the audit team was told that since they had moved to a ‘just-in-time’ environment, forecasting was no longer important. None of these companies, to date, has made any changes in sales forecasting management and, thus, has not realized any of the supply chain cost savings and customer service benefits achieved at the other companies. Schultz (1984) cites a number of factors that influence the success or failure of organizational efforts to implement new forecasting models. One of his findings is that the presence of top management support is the top ranked predictor of implementation success, and lack of top management support is the top ranked predictor of implementation failure. In the examples cited above, top management support was clearly present in the ‘address the problems’ response profile, and top management support was clearly lacking in the ‘assign the blame’ and ‘why should I care?’ responses. Obtaining such top management support is one of the key contributions from a forecasting champion (Mentzer et al., 1997). Consistent with Schultz, one of our conclusions from this auditing research is that the existence of a sales forecasting champion, along with the top management support for forecasting improvement such an individual can obtain, is critical to long-term organizational success in sales forecasting management. Other factors noted by Schultz (1984) that enhance the successful implementation of forecasting models can also be considered in light of reactions to forecast audit recommendations. For example, Schultz cites performance, or the impact of a new

system on managers’ job performance. This factor is consistent with the Mentzer et al. (1999) framework that stage four companies provide performance rewards to all personnel involved in the forecasting process. Similarly, Schultz (1984) mentions that goal congruence positively affects implementation success, while Mentzer et al. (1999) encourage common goal setting through communication, coordination, and collaboration. Thus, while the Schultz paper provides guidance for successful implementation of new forecasting models, many of his points are also consistent with those companies who have demonstrated a positive (‘address the problem’) response to forecast audit findings and recommendations.

4. Implications for practitioners One important finding that emerged from this research is the realization that—as with other types of audits—an outside, unbiased analysis is critical to the success of the audit. Several companies in the study had attempted forecast process improvements on their own, prior to the audit study, and reported frustration with their inability to effect significant organizational change. In these companies, the use of external auditors was very helpful both in the articulation of the company’s true forecasting process and in inspiring management action. Individuals who are directly affected by a company’s forecasting processes will share their experiences and frustrations more freely with external auditors whom they perceive to be unburdened with preconceived ideas and free from any political agendas. This perspective helps to uncover elements of the forecasting process that may not be evident to those who are involved in the process day-to-day. This research reinforces Armstrong’s (1988) call for the value of sales forecasting auditing. Academics and practitioners should develop the skills and knowledge base necessary to conduct such sales forecasting management audits. The positive results companies obtained from the sales forecasting audits conducted to date provide encouragement for other companies to follow suit. The example provided earlier of one company

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

realizing substantial savings in purchasing costs during the first year as a result of more accurate and credible forecasts provides an exemplar. Dramatic savings in supply chain costs are typical when the audit recommendations are fully implemented. From the 16 companies that constitute the database for development of the sales forecasting audit process, there are a number of lessons from which all managers can benefit. Firstly, it is clear from this research that forecasting is a distinct and critical management function, and not just an exercise in technique or software selection. Technique development and selection have been the focus of much research, and this literature has made an enormous contribution to improving sales forecasting accuracy. However, this audit research has demonstrated that companies must look beyond techniques and software, and must pay close attention to overall management of the sales forecasting process. A further lesson this sales forecasting audit research can provide managers is that the four dimensions of forecasting management articulated in Mentzer et al. (1999) are a useful diagnostic and prescriptive framework to affect sales forecasting improvement. A significant portion of the audit methodology described here makes use of this framework, and it has been very helpful for characterizing a company’s current forecasting management status, as well as showing managers the ‘should-be’ state to which they can aspire. Finally, the benchmarking phases of ‘as-is’, ‘should-be’, and ‘way-forward’ developed by managers involved in this auditing research are an excellent way to examine the process of continuous improvement in sales forecasting management. These three phases have provided managers with a clear, concise way to think about the process of continuous improvement, not only in sales forecasting, but also in other business functions and processes. Without both a clear understanding of how a company currently operates (the ‘as-is’), and a vision of what world-class really is (the ‘should-be’), changes to core processes (the ‘way-forward’) will be unfocused and ineffective. Further development, explication, and exploration of the characteristics and nature of these three phases are left to future forecasting research.

23

5. Implications for future research The process and framework described in this article address an important area of forecasting research called for by Armstrong, Brodie, and McIntyre (1987), and followed by Armstrong (1987, 1988) and Fildes and Hastings (1994). It offers a foundation to direct future investigations of forecasting best practices and how the audit described may be used to help organizations improve forecasting. Figs. 1–4 offer a framework of benchmark criteria based on an original in-depth study of 20 leading companies. The criteria were validated and the audit process developed as a result of subsequent studies with 16 additional organizations. It is important that researchers continue to evaluate and improve upon the criteria and process presented. The experience and perspectives of others who use the criteria and process in research settings can help to establish a richer understanding of its applicability in different industry and organizational environments, and under different operating conditions. By expanding the diversity of conditions under which the audit process is tested, researchers will be able to assess better the generalizability of the current criteria and process, and identify areas for improvement. Research may identify circumstances where characteristics presented in the benchmark criteria of Figs. 1–4 cannot be assessed or do not have an impact on forecasting performance. There may also be new criteria which can be added to one or more of the four dimensions of forecasting that will help lead to improved performance. In particular, the exemplars of Armstrong (1987) and Fildes and Hastings (1994) that are presented in Table 2 but not included in the Mentzer et al. (1999) framework should be considered for incorporation into future sales forecasting management audit research. The ultimate goal of this research program should be to develop quantitative measures of the attributes articulated in Figs. 1–4. We encourage other researchers to adapt and utilize this audit methodology and report the results in future research. Future research must also investigate ways to quantify better the impact of changes in sales forecasting practices. As noted by Schultz (1992), to determine if an organization is better off having

24

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25

implemented a new forecasting technique, ‘we must go beyond measures of accuracy and look to objective performance measures such as sales, costs, and profits’ (p. 410). To do so, forecasting research must investigate the attitudes and behaviors that influence the implementation of forecasts in practice. By understanding the relationship between the four dimensions of forecasting management and their impact on the application of forecasts, organizations will be able to apply cost / benefit analysis when determining the most effective allocation of resources to forecasting.

References Armstrong, D., Gosling, A., & Weinman, J. (1997). The place of inter-rater reliability in qualitative research: an empirical study. Sociology, 31 (3), 597–606. Armstrong, J. S. (1987). The forecasting audit. In Makridakis, S., & Wheelwright, S. C. (Eds.), The handbook of forecasting. New York: John Wiley, pp. 584–602. Armstrong, J. S. (1988). Research needs in forecasting. International Journal of Forecasting, 4, 449–465. Armstrong, J. S., Brodie, R. J., & McIntyre, S. H. (1987). Forecasting methods for marketing. International Journal of Forecasting, 3, 355–376. Arter, D. R. (1989). Evaluate standards and improve performance with a quality audit. Quality Progress, 22 (9), 41–43. Chen, F., Drezner, Z., Ryan, J. K., & Simchi-Levi, D. (1999). The bullwhip effect: managerial insights on the impact of forecasting and information on variability in a supply chain. In Quantitative models for supply chain management. Boston, MA: Kluwer Academic. Churchill, Jr. G. A., Ford, N. M., & Walker, Jr. O. C. (1993). In 4th ed, Sales force management. Homewood, IL: Richard D. Irwin. Closs, D. J., Oaks, S. L., & Wisdo, J. P. (1989). Design requirements to develop integrated inventory management and forecasting systems. In Annual conference proceedings. St. Louis, MO: Council of Logistics Management. Cox, J. E. (1987). An assessment of books relevant to forecasting in marketing. International Journal of Forecasting, 3, 515– 527. Cox, J. E. (1989). Approaches for improving salespersons’ forecasting. Industrial Marketing Management, 18, 307–311. Dew, J. R. (1994). The critical role of auditing in continuous improvement. National Productivity Review, 13, 417–422, Summer. Fildes, R., & Hastings, R. (1994). The organization and improvement of market forecasting. Journal of the Operations Research Society, 45 (1), 1–16. Fildes, R., & Ranyard, J. C. (1997). Success and survival of operational research groups—a review. Journal of the Operational Research Society, 48 (4), 336–360.

Fliedner, G. (2001). Hierarchical forecasting: issues and use guidelines. Industrial Management and Data Systems, 101 (1), 5–12. Galfond, G., Ronayne, K., & Winkler, C. (1996). State-of-the-art supply chain forecasting. PW Review, 1–12, November. Griggs, S. (1987). Analysing qualitative data. Journal of the Market Research Society, 29 (1), 15–34. Hughes, M. A., & Garrett, D. E. (1990). Intercoder reliability: estimation approaches in marketing: a generalizability theory framework for quantitative data. Journal of Marketing Research, 27, 185–195, May. Hussey, D. E. (1995). Human resources: a strategic audit. International Review of Strategic Management, 6, 157–195. Kahn, K. B. (1998). Revisiting top-down versus bottom-up forecasting. Journal of Business Forecasting, 14–19, Summer. Kahn, K. B., & Mentzer, J. T. (1994). The impact of team-based forecasting. Journal of Business Forecasting, 18–21, Summer. Kahn, K. B., & Mentzer, J. T. (1996). EDI and EDI alliances: implications for the sales forecasting function. Journal of Marketing Theory and Practice, 72–78, Spring. Kurasaki, K. S. (2000). Intercoder reliability for validating conclusions drawn from open-ended interview data. Field Methods, 12 (3), 179–194. Marien, E. J. (1999). Demand planning and sales forecasting: a supply chain essential. Supply Chain Management Review, 76–86, Winter. Makridakis, S., & Wheelwright, S. (1977). Forecasting: issues and challenges. Journal of Marketing, 24, 24–38. McIntyre, S. H., Archabal, D. D., & Miller, C. M. (1993). Applying case-based reasoning to forecasting retail sales. Journal of Retailing, 69, 372–398. Mentzer, J. T. (1988). Forecasting with adaptive extended exponential smoothing. Journal of the Academy of Marketing Science, 16, 62–70. Mentzer, J. T. (1999). The impact of forecasting improvement on return on shareholder value. Journal of Business Forecasting, 7–9, Fall. Mentzer, J. T., & Bienstock, C. C. (1998). Sales forecasting management. Thousand Oaks, CA: Sage. Mentzer, J. T., Bienstock, C. C., & Kahn, K. B. (1999). Benchmarking sales forecasting management. Business Horizons, 48–56, May–June. Mentzer, J. T., & Cox, J. E. (1984a). Familiarity, application, and performance of sales forecasting techniques. Journal of Forecasting, 3, 27–36. Mentzer, J. T., & Cox, J. E. (1984b). A model of the determinants of achieved forecast accuracy. Journal of Business Logistics, 5 (2), 143–155. Mentzer, J. T., & Gomes, R. (1994). Further extensions of adaptive extended exponential smoothing and comparison with the M-competition. Journal of the Academy of Marketing Science, 22, 372–382. Mentzer, J. T., & Kahn, K. B. (1995). Forecasting technique familiarity, satisfaction, usage, and application. Journal of Forecasting, 14, 465–476. Mentzer, J. T., & Kahn, K. B. (1997). State of sales forecasting systems in corporate America. Journal of Business Forecasting, 16, 6–13.

M. A. Moon et al. / International Journal of Forecasting 19 (2003) 5–25 Mentzer, J. T., & Kent, J. L. (1999). Forecasting demand in the Longaberger Company. Marketing Management, 46–50, Summer. Mentzer, J. T., Moon, M. A., Kent, J. L., & Smith, C. D. (1997). The need for a forecasting champion. Journal of Business Forecasting, 16, 3–8, Fall. Mentzer, J. T., & Schroeter, J. (1993). Multiple forecasting system at Brake Parts, Inc. The Journal of Business Forecasting, 12, 5–9, Fall. Mentzer, J. T., & Schroeter, J. (1994). Integrating logistics forecasting techniques, systems, and administration: the multiple forecasting system. Journal of Business Logistics, 13 (2), 205–225. Miller, D. M. (1985). Anatomy of a successful forecasting implementation. International Journal of Forecasting, 1, 69– 75. Moon, M. A., Mentzer, J. T., Smith, C. D., & Garver, M. S. (1998). Seven keys to better forecasting. Business Horizons, 44–52, September–October. Nelson, P. T. (1987). Viewpoint: a forecast is not a sales plan. Journal of Business Logistics, 8 (2), 115–122. Sanders, N. R., & Manrodt, K. B. (1994). Forecasting practices in US corporations: survey results. Interfaces, 24 (2), 92–100. Schultz, R. (1984). The implication of forecasting models. Journal of Forecasting, 3 (1), 43–55. Schultz, R. (1992). Fundamental aspects of forecasting in organizations. International Journal of Forecasting, 7, 409–411. Smart, R. (1995). Forecasting: a vision of the future driving the supply-chain of today. Logistics Focus, 3 (8), 15–16. Smith, S. A., McIntyre, S. H., & Achabal, D. D. (1994). A two-stage sales forecasting procedure using discounted least squares. Journal of Marketing Research, 31, 44–56, February. Tybout, A. M., & Hauser, J. R. (1981). A marketing audit using a conceptual model of consumer behavior: application and evaluation. Journal of Marketing, 44, 82–101, Summer. Waddell, D., & Sohal, A. S. (1994). Forecasting: the key to managerial decision making. Management Decision, 32 (1), 41–49. Winklhofer, H., Diamantopoulos, A., & Witt, S. F. (1996). Forecasting practice: a review of the empirical literature and an agenda for future research. International Journal of Forecasting, 12, 193–221.

25

Wright, D. J. (1988). Decision support oriented sales forecasting methods. Journal of the Academy of Marketing Science, 16 (4), 71–78. Biographies: Mark A. MOON is an Associate Professor at the University of Tennessee, Knoxville. He earned his BA and MBA from the University of Michigan, and his Ph.D. from the University of North Carolina at Chapel Hill. Dr. Moon’s professional experience includes positions in sales and marketing with IBM and Xerox. He has published in the Journal of Personal Selling and Sales Management, Business Horizons, Journal of Business Forecasting, Industrial Marketing Management, Journal of Marketing Education, Marketing Education Review, and several national conference proceedings. Dr. Moon also serves on the editorial review board of the Journal of Personal Selling and Sales Management. John T. (Tom) MENTZER is the Harry J. and Vivienne R. Bruce Excellence Chair of Business Policy in the Department of Marketing, Logistics and Transportation at the University of Tennessee. He has published more than 140 articles and papers in the Journal of Forecasting, Journal of Business Logistics, Journal of Marketing, Journal of Business Research, International Journal of Physical Distribution and Logistics Management, Transportation and Logistics Review, Transportation Journal, Journal of the Academy of Marketing Science, Columbia Journal of World Business, Industrial Marketing Management, Research in Marketing, Business Horizons, and other journals. Carlo D. SMITH is an Associate Professor of Marketing at the University of San Diego. He holds a BS and MBA in logistics management from the Pennsylvania State University and received his Ph.D. in logistics and marketing from the University of Tennessee. His articles have appeared in the Journal of Business Logistics, Journal of Business Forecasting, Business Horizons, and the Journal of Consumer Satisfaction, Dissatisfaction and Complaining Behavior. Before attending to doctoral studies, he spent 12 years in industry as a logistics consultant, executive educator, and corporate logistics manager.