TOTAL QUALITY MANAGEMENT, VOL. 13, NO. 1, 2002, 123 - 131

Measuring the dimensions of quality in higher education Stanley M. Widrick,1 Erhan Mergen1 & Delvin Grant2 1

College of Business, Rochester Institute of Technology, 108 Lomb Memorial Drive, Rochester, NY 14423, USA & 2School of Accountancy and MIS, DePaul University, 1 East Jackson Blvd, Chicago, IL 60604, USA

abstract This paper deals with measuring the three quality dimensions (quality of design, quality of conformance and quality of performance) in higher education. The proposed framework discussed in this paper builds on the model developed by Mergen et al. (2000, Total Quality Management, 11, pp. 345- 352). It discusses a set of measurement parameters to be used in evaluating the quality of research and curriculum development and the tools/techniques necessary for evaluating them. Two examples are discussed of how Rochester Institute of Technology has used this approach to plan and implement a new MS degree and a new research reward system. Introduction The application of quality management (QM) principles to solve industry-related problems has been institutionalized at many ® rms. QM is widespread and heavily institutionalized in companies such as Xerox, Motorola and Tennessee Valley Authority. It is part of the organizational culture and permeates all aspects of the daily organizational life of management and workers. Several universities have used QM as a means to gain competitive advantage (Montano & Utter, 1999; Swift, 1996) or to improve organizational performance (Kanji & Tambi, 1998; Koch & Fisher, 1998; Muse & Burkhalter, 1998). In most of these studies, a QM framework was missing and, more speci® cally, quality of performance was not measured (Grant et al., 1999). We believe that the formal use of a framework would bring discipline to the QM implementation process and bene® t the continuous improvement eþ orts. The QM framework mentioned above is comprised of three dimensions: quality of design (QD), quality of conformance (QC) and quality of performance (QP). It builds on earlier work by Mergen et al. (2000). The objective of this paper is to propose a set of measures and tools to evaluate each of the three components of quality with respect to research and curriculum development in higher education. However, the paper does not deal with the teaching-related activities. This is a research topic that warrants it own discussion and will be dealt with in a separate paper. The remainder of the paper is organized as follows. The second section discusses the three components of quality management described by Mergen et al. (2000). The third section lists a set of proposed measures and tools/techniques for measuring them. The fourth Correspondence: S. Widrick, College of Business, Rochester Institute of Technology, Rochester, NY 14623, USA. E-mail: [email protected] ISSN 0954-4127 print/ISSN 1360-0613 online/02/010123-09 DOI: 10.1080/09544120120098609

© 2002 Taylor & Francis Ltd

124

S. M. WIDRICK ET AL.

section presents two case studies; one examines curriculum development and the other discusses a program to increase research output at the college of business at Rochester Institute of Technology. The paper ends with a discussion/conclusion. TQM measurement framework Mergen et al. (2000) propose that the parameters of quality can be grouped into three areas: (1) quality of design; (2) quality of conformance; and (3) quality of performance. An overview of the model is shown in Fig. 1. This ® gure suggests that there is a logical ¯ ow from QD to QC and QP. However, there are situations when QC may be a design issue and one may be forced to return to the QD phase. The proposed model, when used in conjunction with proper measurement tools, may help colleges improve the quality of their curriculum and research development activities. Quality of design Quality of design has to do with how well the design captures the consumer’s requirements. It is determined by three factors: (1) the quality of the insights gained about the consumer and the depth of understanding of their requirements; (2) the quality of the process used to translate these consumer insights and requirements into a product and/or service that provides value to the consumer; and (3) the continuous improvement of the design process. Quality of conformance Quality of conformance deals with how well the design requirements are satis® ed, including the uniformity, dependability and cost requirements. Quality of conformance is determined by the minimization of variance from design requirements for the goods and/or services. Thus, for each design speci® cation, a proper measure or measures should be developed in

Figure 1. TQM measurement framework.

MEASURING THE DIMENSIONS OF QUALITY

125

order to make sure that design requirements are being met. This is relatively easy in manufacturing operations since the output is tangible and easy to measure. In higher education there are at least two reasons why measurement is a challenging task. First, the service being rendered is not tangible. Second, there is a long lead-time between delivery of the service and the realization of the rewards or bene® ts. Quality of performance Quality of performance deals with how well a service and/or product performs in the eyes of the end-user or consumer (both internal and external). However, without regard to whether the customer is internal or external, performance measurement is always external to the organization being measured. Proposed measures, tools and techniques Listed in Tables 1- 3 is a set of proposed measures and tools that could be used for curriculum development and research. The list is not exhaustive but is a subset of possible tools and dimensions that may be used. For descriptions of these tools and their limitations see Tague (1995). Case studies Curriculum development: MS in Manufacturing Management and Leadership This case deals with curriculum development of an MS in Manufacturing Management and Table 1. Quality of design Dimensions Curriculum What courses to oþ er and sequencing What programs to oþ er Determining appropriate faculty quali® cations Admission requirements Course content

Research Internal commitment and Support (e.g. course release time, money) External support (e.g. research projects, equipment, funding) Accrediting bodies’ (AACSB, State) guidelines Faculty/professional norms and expectations Number and type of research centers of excellence

Measurement tools Survey, tree diagram, ¯ ow diagram, benchmarking, QFD Survey, tree diagram, benchmarking, QFD Benchmarking, matrix of accrediting institutions guidelines, con® rmation checksheet Benchmarking, matrix of accrediting institutions guidelines, con® rmation checksheet Benchmarking, matrix of accrediting institutions like AACSB guidelines, QFD, aý nity diagram, ¯ ow diagram Benchmarking, depth interviews, survey to determine expectations Benchmarking, depth interviews, survey to determine expections Operational de® nitions of expectations of the various bodies Operational de® nitions of faculty/administration expectations Benchmarking, tree diagram to determine success characteristics

126

S. M. WIDRICK ET AL.

Table 2. Quality of conformance Dimension examples

Measurement tools

Curriculum Coverage of the topics (i.e. are the topics listed in the curriculum covered?) Level of coverage Are the courses oþ ered in proper sequence? Do quali® ed instructors teach the courses? Research Papers submitted (over the years) Funding proposals submitted (over the years) Funding proposals accepted (over the years) Papers accepted (over the years) Suppor t for research versus the yield Suppor t for research over the years Distribution of the accepted papers by the journals (i.e. top tier, second tier, etc.)

Con® rmation check sheet (i.e. yes/no-type tally) Peer review, student feedback Con® rmation check sheet (i.e. yes/no-type tally) Con® rmation check sheet (i.e. yes/no-type tally) Trend analysis (e.g. per faculty, per program), control chart Control charts (diþ erent kinds depending on the type and availability of the data), trend analysis Same tools as mentioned above Same tools as mentioned above Scatter diagram Trend analysis Matrix diagram Pie charts Pareto charts

Table 3. Quality of performance Dimension examples Curriculum Employer satisfaction with coop or interns Employer satisfaction with ® nal placement Type of employers recruiting for majors in your curriculum Starting salaries of alums Assess alumni positions and compensation over their career cycle Ability to achieve on standardized tests or licensing boards Research Papers accepted following a blind refereed process Causes for rejection of papers Meeting the scholarship Standards of accrediting bodies No. of and dollar amounts of funded research

Measurement tools Focus group analysis, depth interview, Pareto analysis to determine areas to improve Focus group analysis, depth interview, Pareto analysis Trend analysis, Pareto analysis, control charts, matrix Trend analysis on median starting salaries Surveys, period histograms on salary distributions, trend analysis on median salaries Bar charts, histograms, trend analysis on average scores Histogram, control chart, trend analysis by publication outlet Pareto analysis, ® shbone Checklists Bar chart, trend analysis

Leadership at Rochester Institute of Technology ( Johnson et al., 1995). It will be presented from a design, conformance and performance point of view. See Table 4 for the suggested measurement parameters and tools for curriculum development. The program was developed as part of RIT’s commitment to the regional manufacturing community made possible by an IBM total quality management (TQM) grant. The program was a joint venture between the College of Business and the College of Engineering. A crossdiscipline design team consisted of faculty from both colleges and a former business executive

MEASURING THE DIMENSIONS OF QUALITY

127

Table 4. Measurement parameters and measurement tools for curriculum development Framework

Measurement parameters

Measurement tools

Quality of design

Market needs

Survey for voice of the customer, QFD, focus group, interviews QFD, focus group

Manufacturing knowledge and skill requirements Emerging educational training program for manufacturing Knowledge and skill requirements of the region Module content Quality of conformance Program design

Course sequence Course content Quality of performance

Program design Course content

Focus group, benchmarking surveys, interviews Focus group, benchmarking surveys, interviews Aý nity diagram, focus group, benchmarking Reviews of industry partners, feedback, con® rmation Checksheet Con® rmation checksheet Student feedback, peer review Review, analysis, industry partners’ feedback Capstone project evaluation industry feedback, post-satisfaction survey, new students from employers

on the staþ at the Center for Integrated Manufacturing Studies. Over a dozen industrial advisors worked with the design team. The team assessed the educational requirements for future manufacturing managers to determine how the colleges might satisfy those needs. The team identi® ed market needs and developed speci® c product ideas. These included knowledge and skill requirements for success in manufacturing, emerging educational training program for manufacturing, and the knowledge and skill requirements of the region. Using a quality functional development (QFD) process, the team translated the `voice of the customer’ into a set of program requirements. This led to the development of the ® rst draft of the proposal that was evaluated by a team of executives and RIT students. Based on the feedback, the program was revised. One important revision was due to feedback that suggested that future careers in manufacturing would be quite diþ erent from those in the past. Companies including Bausch & Lomb, Fisons, Harris, IBM, ITT, Kodak and Xerox were consulted to revise the program and its requirements. This led to 143 requirements that were consolidated into 37 learning modules using the aý nity diagram and priority ranking. Advisors evaluated the importance of each module using a budgeted voting procedure. The team then examined issues of sequencing and time constraints related to the various modules. Syllabuses and learning objectives were developed. To verify that the program design met the customer requirements the advisors were brought back to review its design and content. This led to changes and another round of reviews. The changes included adding the word leadership to the program title, moving a quality and statistics course up front and reinstating a module on manufacturing costs. Conformance was also measured by soliciting feedback from students during the time the program was being delivered. An external reviewer, with credentials in engineering and business and extensive knowledge of manufacturing, was identi® ed and provided evaluation. His review of the program was favorable. Nevertheless, he suggested three areas of program improvement that were adopted. Performance was also measured by looking at capstone project results, annual

128

S. M. WIDRICK ET AL.

reviews of the program by students and advisors and through the alumni program for example, such as job placement and post satisfaction survey of alumni. Despite the obvious success of the program, several lessons were learned. First, some team members possessed a narrow perspective and this sometimes interfered with the process. Nevertheless, focusing on the customer requirements worked in our favor to minimize this eþ ect. Second, industry and academia have cultural diþ erences and this led to the alienation of one team. Third, it was diý cult to design a top-down program in the absence of a satisfactory framework. The bottom-up market-driven approach helped to overcome some of the problems by keeping us focused on the customers’ requirements. On the positive side there is strong evidence to suggest that the QFD process was largely responsible for the success of the program. It prevented con¯ ict, fostered cooperation toward a common goal and focused the team’s energy on satisfying the customer. It is believed that in the absence of the QFD process the success of the development of the program would have been severely jeopardized. Another positive outcome was the sense of partnership that existed between academia and industry and the ownership for the program that resulted. This has resulted in important customer- supplier relationships between RIT and industry. Finally, the collaborative process enabled the growth of knowledge and understanding of one another’ s discipline and the integration issues that exist between them. For other examples of using QFD in higher education see Burgar (1994) and Ermer (1995). The performance measures (i.e. quality of performance) are somewhat lacking in this program, though there are attempts to rectify the situation. Program team does a formal feedback session at the time of graduation with alumni, their employers and with the faculty who have taught in the program. This feedback is used to make modi® cations in the curriculum, courses and sequencing. As of this time, there has been no formal recording of these data. The team is considering evaluating the value and the impact of the capstone project in students’ organization, along with preparing a post-graduation satisfaction survey for both alumni.

Research productivity enhancement through `pay-for-performance’ This case deals with the goal of increasing faculty scholarship among a business college faculty. Here again, this discussion is presented from a design, conformance and performance point of view (see Table 5). In the fall of 1997, a task force was formed by the Dean to determine realistic strategies for increasing the research output in the College of Business at RIT. The main reason for this initiative was that an American Academy of Collegiate Schools of Business (AACSB) Review Report indicated that the faculty is expected to increase their research productivity. Speci® cally, it recommended increased output in basic research and publication in highquality journals. The task force looked at the de® nition of basic research given by AACSB and examined the potential causes for the lack of research output. Going through the literature and the feedback gathered from the faculty generated a long list of potential causes. Some of these causes were related to the College’ s culture; others were resource related. The committee concluded that the primary causes of the low research output in the College were due to insuý cient reward, high teaching load, increasing ratio of adjunct and visiting faculty and lack of payoþ from recent investments in summer research grants. Based on these ® ndings the task force agreed that the recommendations should be based on the following principles: (1) Pay-for-performance.

MEASURING THE DIMENSIONS OF QUALITY

129

Table 5. Measurement parameters and measurement tools for research enhancement Framework

Measurement parameters

Measurement tools

Quality of design

Internal commitment and support to design a modi® ed reward and recognition system Revision of tenure and promotion norms and expectations Creating an intellectual climate in the College Using the COB/RIT resources more eþ ectively

Benchmarking, survey of faculty, operational de® nitions of basic research of AACSB Survey, brainstorming

Is the new reward system in place Perceived fairness of reward systems Recognition of scholarly work other than monetary reward Revision of tenure and promotion norms Steps taken to improve intellectual climate of College Plan to generate more resources for the faculty and to improve the eþ ectiveness of the existing resources

Con® rmation checksheet Con® rmation checksheet Con® rmation checksheet

Change in research output Quality of research output Meeting AACSB standards Attracting new faculty

Checksheets, trend analysis Bar chart journal types Con® rmation checksheet Matrix analysis (e.g. number and quality of publications)

Quality of conformance

Quality of performance

Brainstorming Brainstorming, faculty survey

Con® rmation checksheet Con® rmation checksheet Con® rmation checksheet

(2) Flexible teaching load. (3) Self-managing careers philosophy. The speci® c recommendations to increase the intellectual climate at the college were: (1) Design of a new rewards and recognition system. (2) Revise tenure and promotion norms. (3) Use COB/RIT resources more eþ ectively. Of these three recommendations only the ® rst has been designed and is in the process of being implemented, therefore we shall concentrate on it. The task force recommended that a new rewards and recognition system should be installed around the principle of `pay-forperformance’ . Faculty who publish refereed journal articles would be rewarded based on their performance. This award, which would be in addition to the annual merit awards, would be one-time pay and/or reduced teaching load for the upcoming year. The amount that would be paid and the reduced teaching load would be determined by the Dean’s oý ce. In general, the higher the quality of research, the higher the rewards would be. The task force also recommended that outstanding research would be recognized in the College through an outstanding scholar award to help improve the intellectual climate. After the pay-for-performance was ® ne-tuned and implemented, the quality of conformance phase began. It has been in place for 2 years. Some measurements that may be used to evaluate its conformance success would be: ·

·

Do we have a fair `pay-for-performance’ reward system in place? Do we have a way to recognize scholarly work?

130

S. M. WIDRICK ET AL.

·

Have the tenure and promotion norms conformed to the basic principles recommended by the task force? · What has been done to improve the intellectual climate of the College? · Do we have a plan to generate more resources for the faculty and to improve the eþ ectiveness of the existing resources? The quality of performance test will be answered in the future by questions such as: ·

Has the research output increased since the inception of this new system? What percent of this increase is attributable to the new system? · Have we satis® ed the AACSB minimum research output requirement for the College? · Has the new system been successful in attracting talented new faculty to the College? ·

Feedback will be used to improve continuously the pay-for-performance system. This may require some design-related changes and/or changes that are related to conformance issues. Table 5 lists some of the tools that were or could be used for a program such as this research enhancement program. Discussion/conclusion We have proposed a set of measurements along with some tools to measure the three dimensions of quality. Many quality management initiatives, especially in service industries, die because we fail in measurement of the outcomes. The proposed framework enables higher education organizations better to implement and measure quality initiatives. The MM&L case discussed earlier used all three (QD, QC, QP) aspects of the TQM measurement framework. An empowered cross-disciplinary faculty team used a QFD process in creating the new curriculum. Several measurement parameters and measurement tools were used, including benchmarking, assessing customer requirements, aý nity diagrams and obtaining systematic customer feedback. The MM&L curriculum team faced a diý culty in the area of program administration. The diý culty was based on the fact that the program was a cooperative eþ ort between two colleges and it lacked coordination. In the initial stages of the project, the administrative structure was ad hoc. Recently this administrative ¯ aw has been addressed by using a more structured QM process. The pay-for-performance case study involved a very diþ erent set of circumstances. For example, an ad hoc team was formed and charged to identify recommendations for improving research productivity. They had no control of the design, development, or implementation. It served only as advisor to the Dean. A number of quality tools were used in the design phase; however, the conformance and performance phases still need to be accomplished. For example, the promotion and tenure norms have not been revised to be consistent with the task force’s recommendations. Nevertheless, the research climate of the College has improved even though we are still early in the process. An important lesson that we learned from these two case studies is the importance of an empowered team. The empowered MM&L team designed and delivered a high-performance curriculum. In contrast, the results of the pay-for-performance team were less impressive. This lack of impressive results is partly due to the fact that they had no input when it came to conformance or performance of their recommendations. It has long been understood in organizations that when you want to improve something, you ® rst must measure it. This is the primary reason we have suggested measurement techniques and tools to measure design quality, conformance quality and performance quality. We have demonstrated how this can be used to improve curriculum development and research productivity. Measurement is an integral part of continuous improvement.

MEASURING THE DIMENSIONS OF QUALITY

131

References Burgar, P. (1994) Applying QFD to course design in higher education, ASQC 48th Annual Quality Congress Proceedings, Las Vegas, NV, pp. 257- 263. Ermer, D.S. (1995) Using QFD becomes an educational experience for students and faculty, Quality Progress, May, pp. 131- 136. Grant, D., Mergen, E. & Widrick, S. (1999) An analysis of quality management approaches in higher education, International Association of Computer Information Systems Proceedings, pp. 70- 76. Johnson, G., Grant, D., Reeve, R., Feckleton, J., Nasr, N. & Wilson, D. (1995) Engineering-business teamwork: the design of a joint Masters Degree program at Rochester Institute of Technology, IBM Annual Sharing Conference, Rochester, NY, May. Kanji, G.K. & Tambi, A.M.A. (1998) Total quality management and higher education in Malaysia, Total Quality Management, 9, pp. 130- 132. Koch, J.V. & Fisher, J.L. (1998) Higher education and total quality management, Total Quality Management, 9, pp. 659- 668. Mergen, E., Grant, D. & Widrick, S. (2000) Quality management applied to higher education, Total Quality Management, 11, pp. 345- 352. Montano, C. & Utter, G. (1999) Total quality management in higher education, Quality Progress, August, pp. 52- 59. Muse, W. & Burkhalter, B. (1998) Restructuring brings quality improvements to Auburn University, Total Quality Management, July. Swift, J. (1996) Using TQM to identify education improvements in the College of Education at the University of Miami, Computers and Industrial Engineering. Tague, N.R. (1995) The Quality Toolbox (Milwaukee, WI, ASQ Quality Press).