How spreadsheet applications affect information quality Saša Baškarada Monash University Clayton VIC 3800 Australia

ABSTRACT Spreadsheet applications, and in particular Microsoft Excel, are now ubiquitous. Even though, many large organisations heavily rely on them for data analysis, management reporting, and decision making, limited research regarding their potential impacts on organisational information quality has been published. This paper aims to bridge that gap in the literature by identifying key factors — inherent to spreadsheet applications as well as related to their use — which may have significant negative effects on information quality in organisations. The findings presented in this paper have been identified as a part of a broader ethnographic study on information quality, which was conducted in a large telecommunications company over a period of six months. This paper shows that the diffusion of spreadsheet applications is driven by reporting limitations inherent in existing transactional and Business Intelligence (BI) systems. However, while the use of spreadsheets may often be justified from the operational perspective, it frequently leads to significant negative effects on the quality of relevant information. Keywords: data quality, information quality, spreadsheet application, ethnography INTRODUCTION End-user computing has been defined as “the autonomous use of information technology by knowledge workers outside of the information systems department” [4, p. 115]. As such, spreadsheets are the most common data analysis and manipulation tools used by end users in organisations [48]. Spreadsheets are often used as tools for modelling relevant for management decision making [1], and most contemporary Business Intelligence (BI) tools allow for integration with Microsoft Excel [42]. What’s more, some organisations even use Excel as their main BI client [7, 22, 42]. For instance, pivot tables are often used for multidimensional analysis since they provide rollup, drill-down, and slice-and-dice functionality [7]. According to Gartner Research, “the critical path to virtually every materially significant enterprise financial statement includes multiple spreadsheets” [20, p. 4], and many financial services companies often use complex spreadsheets to price a range of financial derivatives [21]. However, in order to maintain the quality of information in organisational information systems, it is imperative to control the processes, which introduce, modify and transform relevant information [11]. Nevertheless, organisations often export critical information from transactional systems to spreadsheets and, thus, separate it from source system information integrity

Received: May 24, 2010

Spring 2011

controls [20]. Such spreadsheets may form significant financial, reputational, as well as regulatory risks [21]. For instance, a trader in a major European bank was recently able to conduct a series of unauthorised trades — leading to $691 million in losses – by editing the spreadsheets used to monitor his unit’s activities [20]. This paper uses the Wang and Strong information quality dimensions framework [51] to analyse data collected during a six month long ethnographic study in order to explore how the use of spreadsheet applications may negatively impact on the quality of organisational information. Relevant factors inherent to spreadsheet applications as well as factors related to their use are identified. In addition, the reasons behind the diffusion of spreadsheet applications in organisations are explored. background Spreadsheet Applications Previous research has shown that most spreadsheets are probably developed by business experts [41]. Since spreadsheet diffusion was initially led by self taught users through internal/ interpersonal channels [4], relevant knowledge is often assumed, and many users are required to acquire it independently [9]. On the other hand, it has been shown that the quality of spreadsheets is dependent on the relevant knowledge of users and developers [31]. Experiments have even suggested that a substantial percentage of spreadsheets created by experienced users contain errors [6]. While previous studies have investigated categories of errors, frequencies of errors, detection of errors, and the impact of errors on spreadsheet results [2, 36-39], there has been limited research on the reasons for errors and their broader impacts [39]. Taylor et al. carried out case studies in 34 UK organisations and found that none of the 34 organisations researched used an information systems methodology for spreadsheet development related tasks [48]. Due to the lack of overall coordination, they found that there was a lot of duplication of effort (and data) with respect to spreadsheet development activities. Furthermore, there was a general lack of spreadsheet related quality assurance activities — for instance, spreadsheet models were rarely tested. In addition, spreadsheets were seldom backed up, and there was a general lack of formal spreadsheet software as well as system development methodology related training. Also, data normalisation and entity modelling were not regarded as relevant by spreadsheet users and developers. Powell et al. estimated the quantitative impacts of errors in 25 operational spreadsheets from five different organisations [41]. They identified 381 potential errors, of which 117 (31%) were

Revised: August 22, 2010 Journal of Computer Information Systems

Accepted: September 28, 2010 77

confirmed as errors by the developers of the spreadsheet. Another study found errors in 0.9% to 1.8% of all formula cells [40]. Information Quality Information quality management is a complex process, which involves a range of cost-benefit tradeoffs [10]. Most organisations depend on it for everyday business operations [16, 23, 45-46, 54], and some organisations are even starting to recognise its potential contributions toward achieving a strategic competitive advantage [12]. As a result, interest in information quality management research and practice is growing globally [14, 30]. Studies have highlighted the importance of information quality to a wide range of domains, including Enterprise Resource Planning (ERP) [19, 49, 52-53], Supply Chain Management (SCM) [24, 32, 43-44], data warehousing [29], advanced data mining/analytics [3, 5, 34], and product data management [47]. In a survey of more than 140 companies in various industries and geographic regions Gartner Research asked respondents to estimate the impact of poor information quality on their Table 1 — Information Quality Dimensions [51] Dimension Description



Believability

Data are accepted or regarded as true, real, and credible.

Accuracy

Data are correct, reliable, and certified free of error.

Objectivity

Data are unbiased (unprejudiced) and impartial.

Reputation

Data are trusted or highly regarded in terms of their source or content.

Value-Added

Data are beneficial and provide advantages from their use.

Relevancy

Data are applicable and helpful for the task at hand.

Timeliness

The age of the data is appropriate for the task at hand.

Completeness

Data are of sufficient breadth, depth, and scope for the task at hand.

Amount

The quantity or volume of available data is appropriate.

Interpretability

Data are in appropriate language and units and the data definitions are clear.

Understandability

Data are clear without ambiguity and easily comprehended.

Consistency

Data are always presented in the same format and are compatible with previous data.

Conciseness

Data are compactly represented without being overwhelming.

Accessibility

Data are available or easily and quickly retrievable.

Access Security

Access to data can be restricted and hence kept secure. 78

organisations. The survey found that poor information quality has a negative financial impact on most organisations, with average estimated annual losses of $8.2 million. Some organisations even indicated annual losses as high as $100 million [13]. Wang and Strong defined information quality as “fitness for use” [51, p. 6], which implies that information quality depends on the user as well as the context in which the information is used. Other studies have shown that user perceptions of information quality may also depend on the source of that information [25]. Accordingly, Wang and Strong defined information quality dimensions from users’ perspective, developing a framework that comprises 15 relevant dimensions (Table 1). These have now become generally accepted in the literature [30]. Methodology Ethnography, which requires the scientist to spend a considerable amount of time in the field, originates from social and cultural anthropology research [33]. As a result, it is particularly well suited to exploring organisational contexts of information systems [8, 33]. Ethnography has been described as “the most in-depth or intensive research method possible” [33, p. 6], and it is actually an umbrella term for a range of different analytic frameworks [8]. Key differences between case study research and ethnography include the amount of time spent in the field as well as the extent of participant observation involved in data collection [33]. Similarities include the fact that both ethnographic as well as case study research lead to theoretical rather then to statistical generalisations [28, 33, 55]. Due to the richness of ethnographic studies, and the “fact that it is impossible to tell the whole story in any one paper” [33, p. 11], Myers recommends publishing parts of such research separately. Accordingly, this paper forms a part of a broader ethnographic study on information quality, which was conducted in a large telecommunications company over a period of six months. The company provides a wide range of services, including fixed-line/mobile telephone, dialup/wireless/DSL/cable Internet, and pay TV, to residential, business, and government customers. Due to its size and market dominance, it also provides wholesale services to other service providers. The company structure is a mixture of functional and divisional configuration. Thus, while it has a range of functional business units (e.g. finance, human resources, and marketing) it also has several major business units that are divided by customer segments (i.e. consumer, business, government, etc.). During a period of six months, the researcher spent more than 1,000 hours at the company, providing information quality/analytics related consulting services to two departments (complaints and risk/compliance) in the consumer business unit. The complaints department dealt with a large amount of complaints related data, which was received directly from customers, or indirectly from government regulators. The risk and compliance department dealt with a wide range of information sets, including information related to marketing, customers, internal training, sales, and the like. Data was collected through participant observations, meetings, interviews, a range of organisational documents, and informal social contacts, from a wide range of employees, including business analysts, project managers, IT analysts/developers, marketing analysts/managers, risk/compliance/privacy coordinators/managers, and the like. Ethnographic data is interpreted through scientist’s theoretical context, and the analysis usually involves the coding of notes,

Journal of Computer Information Systems

Spring 2011

lead to problems with accuracy. The subsequent section discusses each of the factors in greater detail.

which are then compared to identify patterns and themes [8]. Following recommendations from literature [33], the researcher continuously wrote up and analysed written notes, thus, producing regular research memos. Furthermore, the following key principles for evaluating interpretive ethnographic research [26, 33] have been addressed by this paper:

Table 2 — Rate of Occurrence

• it presents a novel contribution to the field; • it offers rich insights; • it includes a significant amount of data; and • it explains in detail how the data was analysed. the Analysis framework

Rare

The issue was encountered only one time during the study.

Occasional

The issue was encountered several times during the study.

Frequent

The issue was encountered on a regular basis during the study.

Table 3 — Impact on Information Quality

As previously mentioned, ethnographic data is interpreted through scientist’s theoretical context. As such, the Wang/Strong framework for information quality presented in the background section formed the theoretical lens, which has been used to analyse the written notes. Additionally, due to the richness of the data, it was required to further restrict the scope of the finding presented in this paper. Thus, only the factors with relatively significant rate of occurrence (Table 2) and impact on information quality (Table 3) were considered relevant (Table 4). results Table 5 summarises the results of data analysis. It shows spreadsheet related factors, which may have a negative impact on one or more information quality dimensions. Positive factors, indicated with a leading (+), imply that their presence may result in a negative impact on information quality. On the other hand, negative factors, indicated with a leading (-), imply that their absence may result in a negative impact on information quality. For instance, the presence of spreadsheet silos may lead to accessibility issues, whereas the absence of integrity controls may

Low

The impact on information quality is negligible. It does not significantly affect the operations of the organisation.

Medium

The impact on information quality is noticeable. It may impact on the efficiency of organisational operations.

High

The impact on information quality is significant. It may impact on the effectiveness of organisational operations. Table 4 — Relevant Factors



Impact on Information Quality

Rate of Occurrence



High

Medium

Low



High

X

X





Medium

X

X





Low

Table 5 — Spreadsheet Related Factors with Negative Impacts on Information Quality

Relevancy

Timeliness

Completeness

Amount

M

M M

M

H

H

M



(+)Redundant Storage

M

M

M

Rate of Occurrence



(+)Manual Data Analysis/Transformations

H

(-)Quality Assurance

H

M

(-)Training

M

H H

H

H

H M

H

H

H M

M

Medium

(-)Configuration Management

(+)Spreadsheet size Limitations



H

H

(-)Security Controls







H

M





H

(+)Spreadsheet Silo High



H

(-)Integrity Controls

(-)Metadata

Spring 2011

Access Security

Value-Added

M

Accessibility

Reputation

(+)Separation of Data from Source Systems

Conciseness

Objectivity



Consistency

Spreadsheet Related Factor

Accuracy

Understandability







Interpretability

Degree of Negative Impact on Information Quality (H: high; M: medium)

Believability



M

M

M

Journal of Computer Information Systems

79

H

Discussion Reasons for Spreadsheet Use This study has identified ad hoc reporting as the main use of spreadsheet applications. As such, data is rarely directly entered into spreadsheets; instead it is usually imported from transactional and Business Intelligence (BI) systems. Even though most of the transactional — and certainly all BI — systems provide reporting functionalities, several reasons why they are not commonly used have been identified (Figure 1): 1. Lack of capital expenditure (CAPEX) funding. 2. The System Development Life Cycle (SDLC) process is too complex. 3. The time frame required for development is too long. Development of new system functionality (such as the development of new reports) is usually considered as capital expenditure and, as such, the requesting business unit first requires the appropriate funding. Any such funding usually has to be formally requested and accompanied by a relevant business case. Given that CAPEX budgets are usually periodically requested and approved, any new reporting requirements would have to be identified well in advance. Furthermore, even if the required funding is approved, the SDLC phases that have to be followed (i.e. analysis, design, implementation, testing) are usually very complex and time consuming. As a result, it may take more than one year to operationalise any new reports. However, ad hoc reports are by definition urgently required and only infrequently used. Consequently, many analysts and managers often prefer to export raw data from source systems and analyse it in spreadsheet applications as required. One of the managers observed: “It takes forever and it costs an absolute fortune to develop these reports. And, once they are developed they are not really what you asked for in the first place anyway.” Additionally, conflicting business priorities may also impact on resourcing required for report development; a business analyst explained: “We had a guy from Oracle developing some reports in APEX, but he got moved to another project before he was able to finish them. So, we don’t have any other choice but to extract the data we need and to analyse it in a spreadsheet.”

Figure 1 — Factors Relating to Spreadsheet Use

what system he’s getting it from, but I think it’s the right data.” As a consequence, such data sets are frequently found to be incomplete as they may not include all relevant information. At the same time, they frequently include much irrelevant information, which may not be value adding. Another manager explained: “We only use about five columns out of 20+ we have in the spreadsheet.” As business experts are usually cautious about deleting data from such spreadsheets, much of the irrelevant information is never removed. This may result in a negative impact on the amount of information found in such spreadsheets. One of the business experts explained: “I do not usually delete any data from spreadsheets unless I’m absolutely sure I won’t need it in the future.” Another issue is that spreadsheet-based data sets are most often exported from source systems at the lowest possible level of granularity. As a result, such raw data may not always be concisely represented. One of the managers commented:

Separation of Data from Source Systems The extraction of information from source systems into spreadsheet applications results in the separation of data from its original context. This may negatively impact believability, objectivity, and reputation of relevant information, since the end users of spreadsheet-based reports/models are not necessarily the same individuals who extracted the data from the source systems. As such, the end users are often unsure about where exactly that information came from, as well as when, how and by whom it was initially extracted. One of the managers explained: “I get my spreadsheets from Jason B. and he gets them from some other guy in his business unit. I am not sure

80

“I am only interested in comparing the performance of different call centres, but the data that I get is at the agent level. I now first have to get someone to aggregate it before I can do anything with it.” Consistency and timeliness may also become an issue if some of the data is updated in the source systems, which is often the case. Another manager observed: “Look, I got this spreadsheet last month and I’m still making decisions based on it. I know that quite a bit of the data has changed in the meantime, but I have no other choice but to keep using it until I get a new spreadsheet.”

Journal of Computer Information Systems

Spring 2011

Redundant Storage Spreadsheets are also regularly shared between stakeholders and, thus, they are often redundantly stored. Taking into consideration that some of the stakeholders are likely to make changes to the original spreadsheet, multiple versions of the truth (i.e. inconsistencies) may emerge. Once business experts realise that spreadsheets are redundantly stored, this may result in a negative impact on the believability, objectivity, and reputation of relevant information. A business expert explained: ”The problem is that too many people have their own copies of this spreadsheet. There is no single version of the truth.” Spreadsheet Silos While it is usually relatively easy to export data from organisational information systems into spreadsheets, it is often much more difficult (frequently impossible) for the data to flow the other way — i.e. to upload spreadsheets to organisational information systems. This limitation frequently results in the creation of spreadsheet silos, which, at the very least, may lead to problems with the accessibility of relevant information. Manual Data Analysis/Transformations

“I haven’t been able to take leave for over a year. . . . No one else knows how to update these spreadsheet models. It’s all in my head.” Business experts often manually transform and analyse spreadsheet-based data sets, resulting in operational inefficiencies as well as potential issues with accuracy, interpretability, and understandability. This study identified a specific example where a business analyst was required to extract numerous spreadsheets, from several different systems, on a daily basis. He would then spend most of the day manually transforming and combining those spreadsheets, so that he could email the required data sets to other business stakeholder before the close of business. The researcher, due to his consulting role within the company, was able to automate most of those steps using Visual Basic for Applications (VBA), so that any manual intervention was minimised and the time required to complete those tasks was reduced from more than five hours to less than one hour. Quality Assurance Manual data analyses and transformations, which may involve numerous formulas and pivot tables, often results in spreadsheet errors (inaccuracies) that are rarely identified due to the general lack of sufficiently rigorous testing. Many ad hoc spreadsheet Spring 2011

“We don’t follow any formal methodologies for spreadsheet development. I guess it would be beneficial to incorporate some formal testing, because I often find errors in the spreadsheets I receive.” Training In line with previous research this study has found that spreadsheet models/reports are frequently developed by business experts who often lack relevant knowledge and skills. Critical knowledge/skill areas that most business experts lack include: software development methodologies, relational data modelling, and scripting — e.g. VBA. Lack of such skills may, at the very least, lead to issues with accuracy, interpretability, and understandability of spreadsheets. Referring to a spreadsheet model he developed, one of the business experts explained: “I’m sure there is a much better way to do this, but this is the only solution I could come up with.”

The use of spreadsheet applications may negatively impact on the accuracy of information in several ways. Specifically, spreadsheet errors may be introduced during manual data entry, data analysis, and data transformations. For instance, spreadsheetbased dashboards/models often comprise numerous spreadsheets, pivot tables, and charts. Such models often quickly become very complex, so that manual updates frequently result in a range of anomalies. A business analyst explained:



reports/models are developed in an ad hoc manner by business experts, who generally do not follow any formal software development methodologies. Besides testing, other quality assurance activities, such as structured walkthroughs, are also rarely performed. This may in turn negatively affect believability, objectivity, and reputation of such data sets. One of the managers explained:

Configuration Management Related to the problem of redundant storage is the issue of configuration management. Given that updates to spreadsheet models frequently result in the creation of many different versions of the same file, version control becomes a key requirement. However, effective configuration management processes are rarely implemented and followed. As a result, different versions of the same model may replicate the same data leading to issues with the amount of information. At the same time, lack of formalised configuration management may lead to issues with believability, objectivity, and reputation of such spreadsheets. One of the managers explained: “Version control is a nightmare. We’ve got so many different versions; everybody has their own. I never know which one is the most up-to-date one. We do try to put the date in the file name, but it’s not the best solution.” Spreadsheet Size Limitations As already mentioned, the file-based nature of spreadsheets may negatively impact on accessibility of the relevant information. For instance, the file size of spreadsheet models and reports can quickly become very big, so that raw data is often deleted, leaving only the analysed/aggregated information. In such cases, the raw data may not be easily accessible. A business analyst explained: “We don’t keep the old data in our model; we only keep the aggregated information. Otherwise the file would be 1GB.” If the raw data is not deleted, and if spreadsheet file size becomes too big, it may not be possible to easily share it within

Journal of Computer Information Systems

81

the organisation. Additionally, some spreadsheet applications restrict the amount of data that they can manage. For instance, the main spreadsheet application used in this telecommunications company limited the number of rows to 65,536. This limitation often had a negative impact on the completeness of information, since many data sets comprised of several hundred thousand rows. In such cases data was sometimes exported into separate spreadsheets. However, in many instances, business experts did not recognise this limitation, thus, not realising that an exported data set was truncated. One of the managers explained: “I only realised that some data was missing because the results were significantly off. I guess I was lucky, otherwise I wouldn’t have noticed.” Metadata Given the fact that spreadsheet-based data sets are separated from their original context — i.e. source systems — and the relevant metadata, interpretability can also be negatively impacted. As such, business experts often have difficulties understanding spreadsheet data because, without the relevant metadata, their units and definitions are not always clear. For instance, data types available in spreadsheet applications may not match the data types from the source systems. Furthermore, data types are often altered during the extraction process — e.g. a numeric field may be exported as text. As a result, data types, definitions and units often have to be inferred from the data sets. A business analyst explained: “There is no documentation. I just have to look at the data and guess what it represents. The column names can sometimes help, but they are often more cryptic than the data itself.” Security Controls Spreadsheets are often emailed and shared over company intranets and even over the Internet. Given that most of such data is at least commercial-in-confidence, emailing spreadsheets over the Internet may pose a significant security risk. Most spreadsheet applications do not allow for user-based authorisations, and any limited access restriction functionalities (e.g. password protection) are rarely used. When spreadsheets are password protected, and if there is a requirement to share such spreadsheets, password management/distribution becomes an issue. Such passwords are often emailed, which also presents a security risk. One of the managers explained:

“When I enter an incorrect value in [ABC system] I get a warning, but I can enter whatever I want into Excel.” concluding remarks This study has extended existing research and made a significant contribution to information quality theory and practice by providing an in-depth explanation of how the use of spreadsheet applications may negatively impact on the quality of organisational information. The Wang/Strong information quality dimensions framework was applied as a theoretical lens to analyse data, which was gathered during a six month long ethnographic study on information quality in a large telecommunications company. The paper found that the diffusion of spreadsheet applications is driven by reporting limitations inherent in existing transactional and Business Intelligence (BI) systems. As such, ad hoc reporting requirements, impacted by financial and time constraints as well as by the complexities of the Software Development Lifecycle (SDLC) process, were identified as the main reason for the use of spreadsheet applications. As a result, data is rarely directly entered into spreadsheets; instead it is usually imported from source systems for analysis. However, separating data from source systems also separates it from its context, relevant business rules, controls (security and integrity) as well as from metadata, which often results in significant negative effects on a range of Wang/Strong information quality dimensions. Additionally, the file-based nature of spreadsheets often places limitations on the size of relevant data sets and may lead to redundant storage or the creation of spreadsheet silos. The general lack of configuration management, relevant training, and quality assurance, as well as manual data analyses and transformations were also identified as key spreadsheet use related factors with negative impacts on information quality. While some of the limitations identified are inherent to spreadsheet applications — e.g. spreadsheet size limitations, the lack of metadata, and the lack of security/integrity controls — it may be possible to overcome some of the other limitations identified in this paper. For instance, relevant training as well as formalised quality assurance and configuration management may potentially lead to significant improvements in information quality. Additionally, automation of any manual analyses and transformations — e.g. through VBA scripting — may also lead to positive benefits. However, as ad hoc reports are urgently required and only infrequently used, it may be more difficult to overcome the main issue identified — separation of data from source systems — as economic feasibility may restrict more formal development of such reports/models. Limitations and Future Research

“We email most of our spreadsheets; there is no other way to get it to others. I do try to password-protect them before emailing, but then I usually also have to email the password as well.” Integrity Controls While spreadsheet-based data sets are usually extracted from transactional systems, the general lack of spreadsheet integrity controls means that additional (inaccurate) data can be manually entered without being validated for data type, format, business rules, and the like. A business expert explained:

82

Several limitations concerning the qualitative ethnographic research methodology applied in this study need to be acknowledged. Accordingly, this section discusses the main limitations identified by the researcher and recommends corresponding future research opportunities. From the positivist perspective, analogous to single case study research [15], it may be argued that ethnographic studies are specific to a particular organisation and, thus, may not be generalisable. Additional potential limitations also include subjective interpretations as well as the inability to control dependant variables and manipulate independent variables, which

Journal of Computer Information Systems

Spring 2011

implies lack of controllability and replicability [27]. However, since information systems phenomena are time, context, and interpretation dependent [18], ethnographic studies are also specific to a particular time, place, and context. Furthermore, ethnography aims towards analytical generalisation, as opposed to statistical generalisation found in quantitative studies. In other words, where statistical generalisation aims to make an inference about a population on the basis of empirical data collected from a sample, analytical generalisation — as applied in this ethnographic study — is made to theory and not to population [33, 50, 55]. Analytical/theoretical generalisation is based on the concept of theoretical sampling, which originated with the development of grounded theory and was used to guide the data collection in this study [17]. In addition, while subjectivity needs to be acknowledged, interpretive research in general seeks to explain phenomena in terms of the meanings they hold for people [35] by starting from the proposition that our knowledge of reality is a social construction by human actors [50]. Nevertheless, as is the case with any research, there is a need to further test the findings identified in this paper in other organisations/contexts. For instance, statistical generalisation of the factors identified in this study could be tested through the application of surveys in other organisations. Finally, while this study has indicated that the use of spreadsheet applications in organisations may result in negative impacts on information quality, there is a need to also investigate any potential impacts of spreadsheet related information quality problems on decision making and organisational operations. acknowledgements The author would like to thank the anonymous reviewers for their constructive comments, which have greatly contributed toward improving the quality of this paper. References [1] Al-Shammari, M. “A Spreadsheet-Based Management Support System for a Product-MIS Optimization Model,” The Journal of Computer Information Systems (39:1), 1998, 43-50. [2] Ballou, D.P., Pazer, H.L., Belardo, S. and Klein, B. “Implications of Data Quality for Spreadsheet Analysis,” ACM SIGMIS Database (18:3), 1987, 13-19. [3] Bose, R. “Advanced Analytics: Opportunities and Chal­ lenges,” Industrial Management & Data Systems (109:2), 2009, 155-172. [4] Brancheau, J.C. and Wetherbe, J.C. “The Adoption of Spreadsheet Software: Testing Innovation Diffusion Theory in the Context of End-User Computing,” Information Systems Research (1:2), 1990, 115-143. [5] Brown, M.L. and Kros, J.F. “Data Mining and the Impact of Missing Data,” Industrial Management & Data Systems (103:8), 2003, 611-621. [6] Brown, P.S. and Gould, J.D. “An Experimental Study of People Creating Spreadsheets,” ACM Transactions on Information Systems (5:3), 1987, 258-272. [7] Chaudhuri, S. and Dayal, U. “An Overview of Data­ Warehousing and OLAPTechnology,” ACM SIGMOD Record (26:1), 1997, 65-74. [8] Crabtree, A., Nichols, D.M., O’Brien, J., Rouncefield, M. and Twidale, M.B. “Ethnomethodologically Informed

Spring 2011

Ethnography and Information System Design,” Journal of the American Society for Information Science (51:7), 2000, 666-682. [9] De Thomas, A., Ray, H.N. and Rowe, R. “The Electronic Spreadsheet: its Use as a Decision Support Tool in Small Firms,” The Journal of Computer Information Systems (31:3), 1991, 38-44. [10] Even, A. and Shankaranarayanan, G. “Utility Cost Perspectives in Data Quality Management,” The Journal of Computer Information Systems (50:2), 2009, 127-135. [11] Fox, T.L., Guynes, C.S., Prybutok, V.R. and Windsor, J. “Maintaining Quality in Information Systems,” The Journal of Computer Information Systems (40:1), 1999, 76-80. [12] Friedman, T. Case Study: Aera Energy’s Comprehensive Focus on Data Quality Generates Competitive Advantage, Gartner Research, 2008. [13] Friedman, T. Findings From Primary Research Study: Organizations Perceive Significant Cost Impact From Data Quality Issues, Gartner Research, 2009. [14] Friedman, T. and Bitterer, A. Magic Quadrant for Data Quality Tools, Gartner Research, 2009. [15] Gable, G.G. “Integrating Case Study and Survey Research Methods: An Example in Information Systems,” European Journal of Information Systems (3:2), 1994, 112-126. [16] Gelle, E. and Karhu, K. “Information Quality for Strategic Quality Planning,” Industrial Management & Data Systems (103:8), 2003, 633-643. [17] Glaser, B.G. and Strauss, A.L. The Discovery of Grounded Theory, Weidenfeld and Nicolson,London, UK, 1967. [18] Guba, E.G. and Lincoln, Y.S. “’Epistemological and Methodological Bases of Naturalistic Inquiry,” Educational Communication and Technology Journal (30), 1982, 233253. [19] Haug, A., Arlbjørn, J.S. and Pedersen, A. “A Classification Model of ERP System Data Quality,” Industrial Management & Data Systems (109:8), 2009, 1053-1068. [20] Heiser, J. Recognize the Risks of Uncontrolled Spreadsheets, Gartner Research, 2005. [21] Heiser, J. and Furlonger, D. Spreadsheet Controls Need a Boost, Gartner Research, 2009. [22] Hoven, J.V.D. “Data Warehousing: Bringing it All Together,” Information Systems Management (15:2), 1998, 1-5. [23] Hwang, M. and Cappel, J.J. “Data Warehouse Development and Management: Practices of Some Large Companies,” The Journal of Computer Information Systems (43:1), 2002, 3-6. [24] Ivert, L.K. and Jonsson, P. “The Potential Benefits of Advanced Planning and Scheduling Systems in Sales and Operations Planning,” Industrial Management & Data Systems (110:5), 2010, 659-681. [25] Klein, B.D. “User Perceptions of Data Quality: Internet and Traditional Text Sources,” The Journal of Computer Information Systems (41:4), 2001, 9-15. [26] Klein, H.K. and Myers, M.D. “A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems,” MIS Quarterly (23:1), 1999, 67-94. [27] Lee, A.S. “A Scientific Methodology for MIS Case Studies,” MIS Quarterly (13:1), 1989, 33-50. [28] Lee, A.S. and Baskerville, R.L. “Generalizing Generali­ zability in Information Systems Research,” Information Systems Research (14:3), 2003, 221-243. [29] Ma, C., Chou, D.C. and Yen, D.C. “Data Warehousing,

Journal of Computer Information Systems

83

Technology Assessment and Management,” Industrial Management & Data Systems (100:3), 2000, 125-134. [30] Madnick, S.E., Wang, R.Y., Lee, Y.W. and Zhu, H. “Overview and Framework for Data and Information Quality Research,” ACM Journal of Data and Information Quality (1:1), 2009, [31] McGilla, T.J. and Klobas, J.E. “The Role of Spreadsheet Knowledge in User-Developed Application Success,” Decision Support Systems (39), 2005, 355-369. [32] Moon, K.L. and Ngai, E.W.T. “The Adoption of RFID in Fashion Retailing: a Business Value-Added Framework,” Industrial Management & Data Systems (108:5), 2008, 596-612. [33] Myers, M.D. “Investigating Information Systems with Ethnographic Research,” Communication of the Association for Information Systems (2:23), 1999. [34] Nemati, H.R. and Barko, C.D. “Key Factors for Achieving Organizational Data-Mining Success,” Industrial Manage­ ment & Data Systems (103:4), 2003, 282-292. [35] Orlikowski, W.J. and Baroudi, J.J. “Studying Information Technology in Organizations: Research Approaches and Assumptions,” Information Systems Research (2:1), 1991, 1-28. [36] Panko, R.R. “What We Know About Spreadsheet Errors,” Journal of End User Computing (10:2), 1998, 15-21. [37] Panko, R.R. and Aurigemma, S. “Revising the Panko– Halverson Taxonomy of Spreadsheet Errors,” Decision Support Systems (49), 2010, 235-244. [38] Powell, S.G., Baker, K.R. and Lawson, B. “An Auditing Protocol for Spreadsheet Models,” Information and Management (45:5), 2008, 312-320. [39] Powell, S.G., Baker, K.R. and Lawson, B. “A Critical Review of the Literature on Spreadsheet Errors,” Decision Support Systems (46), 2008, 128-138. [40] Powell, S.G., Baker, K.R. and Lawson, B. “Errors in Operational Spreadsheets,” Journal of Organizational and End User Computing (21:3), 2009, 24-36. [41] Powell, S.G., Baker, K.R. and Lawson, B. “Impact of Errors in Operational Spreadsheets,” Decision Support Systems (47), 2009, 126-132. [42] Sallam, R.L., Hostmann, B., Richardson, J. and Bitterer, A. Magic Quadrant for Business Intelligence Platforms, Gartner Research, 2010.



84

[43] Sari, K. “Inventory Inaccuracy and Performance of Collaborative Supply Chain Practices,” Industrial Management & Data Systems (108:4), 2008, 495-509. [44] Saura, I.G., Frances, D.S., Contrı´, G.B. and Blasco, M.F. “Logistics Service Quality: a New Way to Loyalty,” Industrial Management & Data Systems (108:5), 2008, 650-668. [45] Shchiglik, C. and Barnes, S.J. “Evaluating Website Quality in the Airline Industry,” The Journal of Computer Information Systems (44:3), 2004, 17-25. [46] Sherer, S.A. and Alter, S. “Information System Risk and Risk Factors: Are They Mostly About Information Systems?,” Communications of the Association for Information Systems (14), 2004, 29-64. [47] Silvola, R. “Product Data Management Practices in HighTech Companies,” Industrial Management & Data Systems (109:6), 2009, 758-774. [48] Taylor, M.J., Moynihan, E.P. and Wood-Harper, A.T. “EndUser Computing and Information System Methodologies,” Information Systems Journal (8:1), 1998, 85-96. [49] Vosburg, J. and Kumar, A. “Managing Dirty Data in Organizations using ERP: Lessons from a Case Study,” Industrial Management & Data Systems (101:1), 2001, 2131. [50] Walsham, G. Interpreting Information Systems in Organisations, John Wiley & Sons,Chichester, 1993. [51] Wang, R.Y. and Strong, D.M. “Beyond Accuracy: What Data Quality Means to Data Consumers,” Journal of Management Information Systems (12:4), 1996, 5-34. [52] Xu, H., Nord, J.H., Brown, N. and Nord, G.D. “Data Quality Issues in Implementing an ERP,” Industrial Management & Data Systems (102:1), 2002, 47-58. [53] Xu, H., Nord, J.H., Nord, G.D. and Lin, B. “Key Issues of Accounting Information Quality Management: Australian Case Studies,” Industrial Management & Data Systems (103:7), 2003, 461-470. [54] Xu, H. and Koronios, A. “Understanding Information Quality in E-Business,” The Journal of Computer Information Systems (45:2), 2004/2005, 73-82. [55] Yin, R.K. Case Study Research, SAGE Publications, Inc,Thousand Oaks, CA, 2008.

Journal of Computer Information Systems

Spring 2011