An Estimation Model for the Savings Achievable by Tool Chains Matthias Biehl, Martin T¨orngren Embedded Control Systems Royal Institute of Technology Stockholm, Sweden {biehl,martin}@md.kth.se

Abstract—Tool chains are sought after by industry due to their promise to improve the productivity of software development by reducing costs. Despite these promises, there are few attempts to quantify costs and productivity improvements achievable with a tool chain. The decision for or against realizing a tool chain design requires a quantitative analysis of the economic benefits achievable with a tool chain. We apply the COCOMO model for cost estimation to create a quantitative model for predicting the cost-savings of tool chains. The cost-savings model can provide support for practitioners and decision makers when facing the decision to create a new tool chain. Keywords-Cost Estimation; Tool Integration; Software Process Improvement

I. I NTRODUCTION Modern software development relies more and more on the use and interplay of sophisticated development tools, but the tools only cover specific parts of the development process or lifecycle in isolation. Most commercial off-the-shelf tools are not designed for easy integration; especially challenging is the integration of tools from different tool vendors and across different disciplines. The latter becomes relevant in multi-disciplinary development [1], such as in embedded software development. The heterogeneity of the product requires specialists from various engineering disciplines, who work on parts of the system. These parts are eventually integrated into a cohesive system, requiring the integration of development tools as well. Many potential improvements to the development process are ascribed to tool chains, such as higher productivity [2] and increased levels of integration and automation [3], which are realized by sharing data across tools, scripting across tools and configuration management across tools. These improvements are difficult to quantify and measure separately. While we acknowledge the importance of all potential improvements, we focus in this paper on the cost savings enabled by the use of appropriate tool chains. The assumption of the scientific community is that tool chains can help to accomplish cost reductions and productivity improvements through increased automation and improved integration between development tools [4]. However,

it is not quantified in which range these improvements can be expected. In this paper we address the question: Can the potential savings introduced by a particular tool chains be quantified and estimated? The approach presented in this paper relies on general software cost estimation models. This paper is structured as follows. In section II we introduce the state of the art in software cost estimation, since it provides the basis for the tool chain cost savings model. An estimation of the cost savings by tool chains is described in section III. We explicitly state the delimitations and limitations of this presented approach in section IV. We compare our approach to related work in section V, mention intended future work and conclude in section VI. II. S TATE OF T HE A RT IN S OFTWARE C OST E STIMATION It is good software engineering practice to estimate the cost of software development before beginning with the implementation. Several techniques exist to determine the cost of software development, see [5] for a literature review. We can distinguish between algorithmic and non-algorithmic methods. Non-algorithmic methods are analogy costing and expert judgment. In analogy costing one compares the project to a similar one with known cost. In expert judgment cost is determined as the average of the opinion of several experts. Algorithmic methods can be empirical or analytical methods. Empirical methods are based on measurements from historical project data. Most algorithmic methods do not calculate cost directly, instead they calculate the development effort as a function of the estimated software size; the relationship between effort and cost is assumed to be linear. Algorithmic methods have a number of parameters that directly or indirectly influence the cost of the software, relating to the product, the project organization, the computer platform, tooling and the skills of programmers. The most important parameter is the size of the artifacts created. The size can be measured in different ways, common units of measurements are lines of code (LOC), function points or object points. Using statistical averages for different programming languages, the size measurements can be

converted to LOC. Function points [6] are calculated from the number of user-input types, user-output types, inquiry types, internal file types and external file types. Object points are used for database applications and are calculated from the amount of screens and reports. A. COCOMO The Constructive Cost Model (COCOMO-II-2000) [7], [8] is a widely used, algorithmic, empirical cost estimation model. Development cost is determined by multiplying the effort estimation by a costfactor. The basic COCOMO model in equation (2) calculates the effort (in person months) as a function over the size estimation. Size is measured in LOC and can be converted from function points according to the tables in the reference manual [7]. The formula has a constant factor A, a proportional effort multiplier M and an exponential scale factor B. Both M and B are composed of several parameters listed in table I, whose values can be determined by the tables provided in the reference manual [7]. cost =ef f ort ∗ costf actor ef f ort =A ∗ size

B

∗M

(1) (2)

A =2.94

(3)

B =0.91 + 0.01 ∗ (P REC + F LEX + RESL + T EAM + P M AT )

(4)

model is based on requirements and design options and the COCOMO post architecture model allows for precise parametrization once an architecture has been established. III. C OST S AVINGS BY T OOL C HAIN U SAGE To determine the cost savings of tool chains in the product development process, we compare the predicted amount of work necessary for developing the product with and without the tool chain. For both predictions we use the COCOMO model. Out of all the parameters of the COCOMO model introduced in section II-A, two parameters are influenced by introducing tool chains. The factor M changes due to increased tool support, measured by the effort multiplier T OOL. Improved tool support is usually not provided by the introduction of the tool chain alone, but the introduction of several tools for lifecycle management, such as a software configuration management system. The factor B changes due to increased process maturity, expressed by the scale factor P M AT : configuring and customizing a tool chain may lead to increased awareness and improvement of the development process. The estimated cost savings introduced by the tool chain are a fraction, called savingf actor, of the original product development cost without the tool chain, as shown in equation 6.

M =RELY ∗ CP LEX ∗ DOCU ∗ DAT A ∗ RU SE ∗ T IM E ∗ P V OL∗ costsavings =savingf actor ∗ costproduct original

ST OR ∗ ACAP ∗ P CON ∗ P CAP ∗ P EXP ∗ AEXP ∗ LT EX∗ T OOL ∗ SCED ∗ SIT E

(5)

Table I PARAMETERS OF COCOMO PREC FLEX RESL TEAM PMAT RELY CPLEX DOCU DATA RUSE TIME PVOL STOR ACAP PCON PCAP PEXP AEXP LTEX TOOL SCED SITE

Previous experience with similar projects Flexibility Extent of risk analysis carried out Cohesion in the team Maturity of the process according to CMMI level Required system reliability Complexity Documentation required Size of database Percentage of reusable components Execution time Volatility of development platform Memory constraint Capability of project analysts Personnel continuity Programmer capability Programmer experience in this domain Analyst experience in this domain Language and tool experience Use of tools Development schedule compression Extent of multi-site working

COCOMO accounts for the fact that more precise cost estimation is possible later in the project, when more information is available. COCOMO actually provides several models that can be used in different development stages: The COCOMO application composition model provides a rough estimate for prototypes, the COCOMO early design

(6)

The savingfactor, called sf , can be calculated by relating the product development cost without a tool chain, costproduct original , to the product development cost with the tool chain, costproduct with T C , as shown in formula 7. Both costs are estimated by the COCOMO model (see equation 2) and since they only differ in the parameters T OOL and P M AT , the formula can be simplified as follows. sf =

costproduct original − costproduct with T C

(7) costproduct original Bproduct with T C Aproduct with T C ∗ size ∗ Mproduct with T C product =1 − Bproduct original Aproduct original ∗ size ∗ Mproduct original product (8) =1 −

T OOLproduct with T C

∗

T OOLproduct original size

0.01∗(P M ATproduct with T C −P M ATproduct original ) product

(9)

In formula 9 for the savingfactor, we can distinguish a size-independent and size-dependent part of sf . In the following sections we isolate the savingfactors sfsize−independent and sfsize−dependent to study them separately by setting the other factor to 1. In a practical setting, both size-independent and size-dependent sf contribute to the total cost savings. A. Size-independent Cost Savings The TOOL parameter in the COCOMO model reflects the level of tool integration and lifecycle support. It is one of the proportional effort multipliers for M . The values of

the T OOL parameter have been obtained empirically in the context of COCOMO [7] and are listed in table II for convenience. The achievable savingf actor for a tool chain depends on the level of tool support that was present before and that exists after the introduction of the tool chain, as described in formula 10. Table III shows the sizeindependent improvements achievable through tool integration and lifecycle support. The rows indicate the level of tool support without a tool chain (T OOLproduct original ), the columns indicate the level of tool support with a tool chain (T OOLproduct with T C ) and the values are the sizeindependent savingfactors in percent. The maximum cost savings of 33% can be achieved when introducing a tool chain with a very high level of integration into an original situation with a very low level of integration. sfsize−independent = 1 −

T OOLproduct with T C

(10)

T OOLproduct original

B. Size-dependent Cost Savings The introduction of tool chains often coincides with a higher awareness of the development process and the introduction of a more rigorous development process. As a result, the level of CMMI (Capability Maturity Model) [9] might be increased, which in turn has an effect on the process maturity scale factor PMAT in the COCOMO model. The effects of an increased PMAT-level on the cost of product development are shown in formula 11. According to the COCOMO model, a higher PMAT-level leads to a decreased product development cost. The savingfactor is dependent on the size of the developed software product. Larger projects have a higher potential for exploiting the CMM process improvements. In addition, there is a higher potential for saving when the gap between the original and the new process maturity is large. Note, that changes in process maturity, especially large ones, are not achievable with tool integration alone; tool integration can merely contribute to and accelerate such an improvement. sfsize−dependent = 1 − size

0.01∗(P M ATwith T C −P M AToriginal ) product

(11)

In figure 1 we visualize the size-dependent savingfactor as a function over the size, based on PMAT values for CMM level 1 (P M ATproduct original ) and CMM level 2 (P M ATproduct with T C ). IV. D ELIMITATIONS AND L IMITATIONS We focus in this paper on the cost savings provided by tool chains and at the same time acknowledge that cost savings are only one of the benefits of tool chains. We explicitly would like to mention that a comprehensive decision making process for the introduction of a tool chain should consider - besides cost savings - other potential improvements and additional costs of tool chains. While productivity improvements, time savings and higher speeds

Figure 1. Size-dependent savingfactor as a function over the size of the product in KLOC

of execution will in the end be reflected by cost savings, other benefits such as quality improvements and increased transparency provided by tool chains are not captured by cost savings and need to be considered separately. Tool chains require additional costs, occurring before and after the introduction of a new tool chain. Before the introduction of the tool chain, there are the development cost of the tool chain. After the introduction, new or enhanced software engineering practices need to be introduced that can deal with the changes that are caused and enabled by tool chains. We explicitly list the limitations of the chosen estimation approach, they pertain to general limitations of estimation approaches and the chosen COCOMO model. We provide a method that not merely measures the cost savings of existing tool chains, but that predicts the theoretically achievable cost savings of future tool chains before they are built; we thus build an estimation model. There is an inherent risk of the accuracy of software cost estimation models, as documented in [10]. Frameworks for measuring the accuracy have been established [11]. Our model inherits the risks of applicability and accuracy from COCOMO, since the estimations in this paper are based on the parameters determined empirically in COCOMO-II-2000 (Constructive Cost Model) [7], [8]. COCOMO-II has been successfully used in practice, and has been calibrated for modern development processes, e.g. for real-time embedded systems development [12]. Despite these facts, we need to be prepared for estimation errors resulting from applying the COCOMO model [13]. V. R ELATED W ORK Tool chains are introduced for a number of reasons, some of them are economical, however, there is not much work on the economic implications of introducing tool chains. According to the extensive literature review and classification of over 300 papers on tool integration [4], [14], there are only two papers on the economic impact of tool integration. Most of the other papers deal with the mechanisms of realizing tool integration. Research on the economic implications of tool chains is suggested in [4] and several economic implications of tool chains are mentioned as possible future work. The authors suggest studying the relationship between the net value of

Table II VALUES OF THE TOOL PARAMETER IN COCOMO [7] TOOL Level Very Low Low Nominal High Very High

Description Edit, code, debug Simple frontend, backend CASE, little integration Basic lifecycle tools, moderately integrated Strong, mature lifecycle tools, moderately integrated Strong, mature, proactive lifecycle tools, well integrated with processes, methods, reuse

Value 1.17 1.09 1.00 0.9 0.78

Table III S IZE - INDEPENDENT COST SAVINGS FACTOR

from Very Low from Low from Nominal From High

to Low 7%

to Nominal 15% 8%

tool integration and its sophistication: The hypothesis is that there is an optimum level of tool integration sophistication, where the net value of the tool chain reaches its maximum. The relationship between the sophistication of tool integration and sustainability has been qualitatively described in [15]. The authors argue that a trade-off between the two exists, i.e. higher sophistication leads to a shorter lifespan of the tool chain. The COCOMO model [16], [8], which we use in this paper, represents tool integration efforts and lifecycle issues one-dimensionally; they are reflected in a single parameter, the TOOL parameter. An extension to the COCOMO model [17] proposes to disaggregate this parameter into three separate ones, namely tool integration, tool maturity and tool coverage. This yields more precise cost estimates. The paper is concerned with the impact of tool integration on the cost of product development, but neither on cost savings by tool chains, tool chain development costs nor on the costefficiency of tool chains. Development with COTS (commercial off-the-shelf) components is similar to building tool chains in so far as they are both compositions of pre-built software. Thus one might suspect that COTS cost estimation models are applicable for tool chains. However, COTS cost estimation models such as COCOTS [18] and others [19] focus on the use of COTS software as part of the delivered product, whereas development tools as COTS software in tool chains are parts of the development environment. This distinction is relevant for the calculation of benefits: the benefits of COTS arise from not having to develop the software, the benefits of tool chains arise from integration and automation. The resulting cost models are thus fundamentally different. COTS models include the cost of the COTS components, however, for tool chains the cost of the integrated development tools should not be included, as they are needed with or without the tool chain. Tool chains are equivalent to what is referred to as ’glue code’ in the COTS community. Some COTS models

to High 23% 17% 10%

to Very High 33% 28% 22% 13%

determine the cost of glue code by an algorithmic method similar to COCOMO and are thus similar to our approach [18], others rely on ’expert judgments’ [19]. The relationship between cost and benefits is widely discussed for COTS models. However, the benefits are usually measured in their own units, rendering trade-off analysis difficult. In this paper we measure the accumulated economic effects of the benefits as cost savings and can thus directly compare them to the development costs. While we have found interest in the net value of tool integration and simple qualitative models, we could, to the best of our knowledge, neither find any work explaining the cost-efficiency for tool chains using a quantitative model, nor on the cost estimation of tool chain development. VI. F UTURE W ORK AND C ONCLUSION In the future we would like to validate the cost-savings model with measured data from industrial development, ideally by comparing product development costs, when developing with and without tool-chains. These measurements can be used to determine the accuracy of the COCOMObased prediction of cost savings. In particular, we would like to compare these measurements with those predicted using the Extended COCOMO [2]. With its three dimensions of describing tool integration, more accurate estimates might be possible. In this work we present an estimation technique for predicting reductions in development efforts and costs through tool chains. This estimation technique can be applied before the tool chain has actually been built. The model can thus support industrial users, when making the decision to introduce a new tool chain. For such a decision many aspects need to be considered, and cost is only one of them, although an important one. The calculations of the savings model can thus provide input for well-informed decision making, especially table III can serve as a first rough estimate of the potential cost savings through tool chains.

R EFERENCES [1] J. El-khoury, O. Redell, and M. T¨orngren, “A Tool Integration Platform for Multi-Disciplinary Development,” in 31st EUROMICRO Conference on Software Engineering and Advanced Applications. IEEE, 2005, pp. 442–450. [Online]. Available: http://dx.doi.org/10.1109/EUROMICRO.2005.10 [2] Y. Yang and J. Han, “Classification of and Experimentation on Tool Interfacing in Software Development Environments,” in Proceedings of the Third Asia-Pacific Software Engineering Conference, ser. APSEC ’96. Washington, DC, USA: IEEE Computer Society, 1996. [Online]. Available: http://ieeexplore.ieee.org/xpls/abs all.jsp?arnumber=566740&tag=1 [3] A. W. Brown and M. H. Penedo, “An annotated bibliography on integration in software engineering environments,” SIGSOFT Softw. Eng. Notes, vol. 17, no. 3, pp. 47–55, 1992. [Online]. Available: http://dx.doi.org/10.1145/140938.140944 [4] M. Wicks and R. Dewar, “A new research agenda for tool integration,” Journal of Systems and Software, vol. 80, no. 9, pp. 1569–1585, Sep. 2007. [Online]. Available: http://dx.doi.org/10.1016/j.jss.2007.03.089 [5] M. Shepperd, “Software project economics: a roadmap,” in Future of Software Engineering, 2007. FOSE ’07. IEEE, May 2007, pp. 304–315. [Online]. Available: http://dx.doi.org/10.1109/FOSE.2007.23 [6] A. J. Albrecht and J. E. Gaffney, “Software Function, Source Lines of Code, and Development Effort Prediction: A Software Science Validation,” IEEE Trans. Softw. Eng., vol. 9, pp. 639–648, Nov. 1983. [Online]. Available: http://portal.acm.org/citation.cfm?id=1313875 [7] B. Boehm, C. Abts, E. Horowitz, A. W. Brown, R. Madachy, S. Chulani, D. Reifer, B. Clark, and B. Steece, “COCOMO II Model Definition Manual,” Center of Software Engineering at USC, Tech. Rep., 2000. [Online]. Available: http://csse.usc.edu/csse/research/ COCOMOII/cocomo2000.0/CII modelman2000.0.pdf [8] B. W. Boehm, C. Abts, A. W. Brown, S. Chulani, B. K. Clark, E. Horowitz, R. Madachy, D. J. Reifer, and B. Steece, Software Cost Estimation with Cocomo II. Prentice Hall, Aug. 2000. [Online]. Available: http: //www.worldcat.org/isbn/0130266922 [9] M. B. Chrissis, M. Konrad, and S. Shrum, CMMI: Guidelines for Process Integration and Product Improvement (Sei Series in Software Engineering), 2nd ed. Addison-Wesley Longman, Amsterdam, Nov. 2006. [Online]. Available: http://www.worldcat.org/isbn/0321279670 [10] K. Molokken and M. Jorgensen, “A review of software surveys on software effort estimation,” in Empirical Software Engineering, 2003. ISESE 2003. Proceedings. 2003 International Symposium on. IEEE, 2003, pp. 223–230. [Online]. Available: http://dx.doi.org/10.1109/ISESE.2003. 1237981

[11] S. Grimstad and M. Jørgensen, “A framework for the analysis of software cost estimation accuracy,” in Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering, ser. ISESE ’06. New York, NY, USA: ACM, 2006, pp. 58–65. [Online]. Available: http://dx.doi.org/10.1145/1159733.1159745 [12] C. Nyborg, “Detailed COCOMO Applied to Real-time Systems at Xerox,” University of Southern California, Tech. Rep., Nov. 1987. [13] M. Korte and D. Port, “Confidence in software cost estimation results based on MMRE and PRED,” in Proceedings of the 4th international workshop on Predictor models in software engineering, ser. PROMISE ’08. New York, NY, USA: ACM, 2008, pp. 63–70. [Online]. Available: http://dx.doi.org/10.1145/1370788.1370804 [14] M. N. Wicks, “Tool Integration within Software Engineering Environments: An Annotated Bibliography,” Heriot-Watt University, Tech. Rep., 2006. [Online]. Available: http://www.macs.hw.ac.uk:8080/techreps/ docs/files/HW-MACS-TR-0041.pdf [15] F. Altheide, “An architecture for a sustainable tool integration,” in Workshop on Tool integration in system development, D¨orr and Schurr, Eds., 2003. [16] B. W. Boehm, Software Engineering Economics, 1st ed. Prentice Hall, Nov. 1981. [Online]. Available: http://www. worldcat.org/isbn/0138221227 [17] J. Baik, B. Boehm, and B. M. Steece, “Disaggregating and Calibrating the CASE Tool Variable in COCOMO II,” IEEE Trans. Softw. Eng., vol. 28, pp. 1009–1022, Nov. 2002. [Online]. Available: http://portal.acm.org/citation.cfm? id=630831.631294 [18] C. Abts, B. W. Boehm, and E. B. Clark, “COCOTS: A COTS Software Integration Lifecycle Cost Model - Model Overview and Preliminary Data Collection Findings,” in In ESCOMSCOPE Conference, 2000. [19] S. Ben, A. B. Lamine, L. L. Jilani, H. Hajjami, and B. Ghezala, “SoCoEMo-COTS: A Software Economic Model for Commercial Off-The-Shelf (COTS) Based Software Development,” pp. 606–612, 2006.