Assessing Vulnerabilities in Apache and IIS HTTP Servers Sung-Whan Woo
Omar H. Alhazmi
Yashwant K. Malaiya
Colorado State University Fort Collins, CO 80523
Colorado State University Fort Collins, CO 80523
Colorado State University Fort Collins, CO 80523
[email protected]
[email protected]
[email protected]
streaming media, mail, etc. An HTTP server has thus emerged as a focal point for the Internet.
ABSTRACT We examine the feasibility of quantitatively characterizing the vulnerabilities in the two major HTTP servers. In particular, we investigate the applicability of quantitative empirical models to the vulnerabilities discovery process for these servers. Such models can allow us to predict the number of vulnerabilities that may potentially be present in a server but may not yet have been found. The data on vulnerabilities found in the two servers is mined and analyzed. We explore the applicability of a timebased and an effort-based vulnerability discovery model. The effort-based model requires the use of the current market-share of a server. Both models have been successfully used for vulnerabilities in the major operating systems. Our results show that both vulnerabilities discovery models fit the data for the HTTP servers well. We also examine two separate classification schemes for server vulnerabilities, one based on the source of error and the other based on severity, and then explore the applicability of the quantitative methods to individual classes.
In this paper we examine the vulnerabilities in the two most widely-used HTTP servers, the Apache server, introduced in 1995, and the Microsoft IIS (Internet Information Services) server, originally supplied as part of the NT operating systems in 1995-96. While Apache has a much larger overall market share, roughly 70%, IIS may have a higher share of the corporate websites. The market share for other servers is very small and thus they are not examined here. IIS is the only HTTP server that is not open-source. Both Apache and IIS are generally comparable in features. IIS runs only under the Windows operating systems, whereas Apache supports all the major operating systems. The security of systems connected to the Internet depends on several components of the system. These include the operating systems, the HTTP servers and the browsers. Some of the major security compromises arise because of vulnerabilities in the HTTP servers. A vulnerability is defined as “a defect which enables an attacker to bypass security measures” [27]. The vulnerabilities found are disclosed by the finders using some of the common reporting mechanisms available in the field. The databases for the vulnerabilities are maintained by organizations such as National Vulnerabilities Database [22], MITRE [19], Bugzilla [6], BugTraq [28] etc., as well as the developers of the software. The exploitations of some of the server vulnerabilities are well known. The Code Red worm [20], which exploited a vulnerability in IIS (described in Microsoft Security Bulletin MS01-033, June 18, 2001), appeared on July 13, 2001, and soon spread world-wide in unpatched systems.
General Terms Security, Measurement
Keywords Vulnerabilities, risk evaluation, quantitative security modeling, HTTP servers.
1. INTRODUCTION There has been considerable discussion of server security in recent years. However, much of this has been qualitative, often focused on detection and prevention of individual vulnerabilities. Quantitative data is sometimes cited, but without any significant critical analysis. Methods need to be developed to allow security related risks to be evaluated quantitatively in a systematic manner. A study by Ford et al. has made a side-by-side comparison between various general servers and the number of vulnerabilities and severity. This study concluded that there is a need to develop some tools for estimating the risks posed by vulnerabilities [12].
All the computing systems connected to the network are subject to some security risk. While there have been many studies attempting to identify causes of vulnerabilities and potential counter-measures, the development of systematic quantitative methods to characterize security has begun only recently. There has been considerable debate comparing the security attributes of open source and commercial software [5]. However, for a careful interpretation of the data, rigorous quantitative modeling methods are needed. The likelihood of a system being compromised depends on the probability that a newly discovered vulnerability will be exploited. Thus, the risk is better represented by the not yet discovered vulnerabilities and the vulnerabilities discovery rate rather than by the vulnerabilities that have been discovered in the past and remedied by patches. Possible approaches for a quantitative perspective of exploitation trends are discussed in [10], [13]. Probabilistic examinations of intrusions have been presented by several researchers [11][18]. In [24], Rescorla has studied vulnerabilities in open source servers. The vulnerabilities discovery process in operating systems has just recently been
Two of the major software components of the Internet are an HTTP (Hyper Text Transfer Protocol) server (also termed a web server) and the browser, which serves as the client. Both of these components were first introduced in 1991 by Tim Berners-Lee of CERN. They have now become indispensable parts of both organizational and personal interactions. The early web servers provided information using static HTML pages. The web server now provides dynamic and interactive services between the server and client using database queries, executable script, etc. The web server is able to support functions such as serving
1
examined by Rescorla [25] and by Alhazmi and Malaiya [1], [2], [3].
share on the software. The second model requires explicit estimation of the effort using an effort measure, which is then used as an independent variable.
Servers are very attractive targets for malicious attacks. Servers can represent the first line of defense that, if bypassed, can compromise the integrity, confidentiality and availability attributes of the enterprise security. Thus, it is essential to understand the threat posed by both undiscovered vulnerabilities and recently discovered vulnerabilities for which a patch has not been developed or applied.
2.1 The Alhazmi-Malaiya Time-Based Model This model, referred to as the Time-Based Model, assumes that the rate of change of the cumulative number of vulnerabilities Ω is governed by two factors, as given in Equation 1 below [1]. The first factor declines as the number of remaining undetected vulnerabilities declines. The other factor increases with the time needed to take into account the rising share of the installed base. The saturation effect is modeled by the first factor. While it is possible to obtain a more complex model, this model provides a good fit to the data, as shown below. Let us assume that the vulnerabilities discovery rate is given by the differential equation:
At this time, despite the significance of security in the HTTP servers, very little quantitative work has been done to model the vulnerabilities discovery process for the servers. Such work would permit the developers and the users to better estimate future vulnerabilities discovery rates. It would also be highly desirable to be able to project what types of vulnerabilities are more likely to be discovered.
dΩ = AΩ( B − Ω) , dt
Some of the available work on HTTP servers discusses some specific problem or attacks that the servers face, such as denial of service attacks (DoS) [8], [14], in which the authors suggests some countermeasures to be applied when an attack of this type takes place. In this paper, our focus is the discovery rates of vulnerabilities of all types.
(1)
where Ω is the cumulative number of vulnerabilities, t is the calendar time, and initially t=0. A and B are empirical constants determined from the recorded data. By solving the differential equation, we obtain
The next section introduces the two vulnerabilities discovery models used. We then consider the total number of vulnerabilities in the two HTTP servers and examine how well the models fit the available data. We then partition the vulnerabilities into categories based on how such vulnerabilities arise, and consider the applicability of the models to individual partitions. As a final step, we partition the vulnerabilities according the severity of impact and again examine the fit provided by the two models. Lastly, we discuss the major observations and present the conclusions.
Ω (t ) =
B BCe
− ABt
+1
,
(2)
where C is a constant introduced while solving Equation 1. Equation 2 gives us a three-parameter model given by the logistic function. In Equation 2, as t approaches infinity, Ω approaches B. Thus, the parameter B represents the total number of accumulated vulnerabilities that will eventually be found.
2. THE VULNERABILTIES DISCOVERY MODELS
A=.013,B=32, C=210 Cumulative Vulerabilities
Use of reliability growth models is now common in software reliability engineering [21]; SRGMs show that as bugs are found and removed, fewer bugs remain. Therefore, the bug finding rate gradually drops and the cumulative number of bugs eventually approaches saturation. Such growth models are used to determine when a software system is ready to be released and what future failure rates can be expected. Vulnerabilities are a special class of defects that can permit circumvention of security measures. Some vulnerabilities discovery models were recently proposed by Anderson [5], Rescorla [25], and Alhazmi and Malaiya [1]. The applicability of these models to several operating systems was examined in [4]. The results show that while some of the models fit the data for most operating systems, others do not fit well or provide a good fit only during a specific phase.
A=.005,B=32 C=60 A=.008, B=32, C=100
Time
Figure 1. Alhazmi-Malaiya Time-Based Model Figure 1 shows hypothetical plots for the time-based model for different values of A, B and C. Thus, the vulnerabilities discovery rate increases at the beginning, reaches a steady rate and then starts declining. Consequently, the cumulative number of vulnerabilities shows an increasing rate at the beginning as the system begins to attract an increasing share of the installed base. After some time, a steady rate of vulnerabilities finding yields a linear curve. Eventually, as the vulnerabilities discovery rate begins to drop, there is saturation due both to reduced attention and a smaller pool of remaining vulnerabilities.
Here, we investigate the applicability of two of the most successful models for HTTP servers. The models used are timebased and effort-based models proposed by Alhazmi and Malaiya [1]. These two models have been found to fit datasets for several of the major Windows and Linux operating systems, as determined by goodness of fit and other measures. The first model considers calendar time as the independent variable. The model incorporates the effect of the rising and declining market
2
commercial system. It should also be noted that the number of vulnerabilities, either found or estimated as remaining, should not be the only measurement of a security threat. Factors such as patch development and application delays and vulnerabilities’ exploitation rates also need to be considered. In this section, all vulnerabilities are considered without regard to how they arise or the extent of their impact. Distinctions among the vulnerabilities will be considered in subsequent sections.
2.2 The Alhazmi-Malaiya Effort-Based Model: Vulnerabilities are usually reported using calendar time, the reason for this is that it is easy to record vulnerabilities and link them to the time of discovery. This, however, does not take into consideration the changes occurring in the environment during the lifetime of the system. A major environmental factor is the number of installations, which depends on the share of the installed base of the specific system. It is much more rewarding to exploit vulnerabilities that exist in a large number of computers. Hence, it can be expected that a larger share of the effort going into the discovery of vulnerabilities, both in-house and external, would go toward a system with a larger installed base.
3.1 Web Server Market Share for the EffortBased Model Market share is one of the most significant factors impacting the effort expended in exploring potential vulnerabilities. Higher market share indicates more incentive to explore and exploit vulnerabilities for both exports and non-exports, since both would find it more profitable or satisfying to spend their time on a software with a higher market share.
Using effort as a factor was first discussed in [10][16]. However, the authors did not suggest a unit or way of measuring effort. The Effort-based Model utilizes a measure termed Equivalent Effort (E), which is calculated using n
E = ∑i=0 (U i × Pi ) = ∑ n
i =0
Ni ,
Table 1. Market Share and Vulnerabilities Found
(3)
where Ui is the total number of all HTTP servers at the period of time i, n represents the last period of usage time, and Pi is the percentage of the servers using the specific server for which we are measuring E. Ni is the number of machines running the specific server during time i. The result is given in systemmonths. The measure E can be calculated for the servers using the data available at [22].
Apache
IIS
SJSWS (SunOne)
Zeus
Other
Market Share
69.7%
20.92%
2.53%
0.78%
6.07%
Vulnerabilities
94
121
3
5
N/A
Release Year
1995
1995
2002
1995
N/A
Latest Version
2.2.0
6.0
6.1
4.3
N/A
Table 1 presents data obtained from NVD and Netcraft, showing the current web server market share and total number of vulnerabilities found to date. As we can see from the table, for servers with a lower percentage of the market, such as Sun Java System Web Server (SJSWS) and Zeus, the total number of vulnerabilities found is low. That does not mean that these systems are more secure, but merely that only limited effort has gone into detecting their vulnerabilities. A significant number of vulnerabilities have been found in both Apache and IIS, illustrating the impact of the market share on the motivation for exploring or finding vulnerabilities. In this study, we use market share as an indicator of effort for the effort-based model.
The model employs equivalent effort as a factor to model vulnerabilities discovery. Equivalent effort reflects the effort that would have gone into finding vulnerabilities more accurately than using time alone. This is somewhat analogous to using CPU time for software reliability growth models (SRGMs)[17]. If we assume that the vulnerabilities detection rate with respect to effort is proportional to the fraction of remaining vulnerabilities, then we get an exponential model like the exponential SRGM. This model can be expressed as follows:
Ω ( E ) = B (1 − e − λ vu E ) ,
Web Server
Figure 2 shows the web server market share for Apache and IIS. As demonstrated by Figure 2 (b), the number of web servers continues to grow steadily. Among the various web servers, Apache and Microsoft IIS dominate the web server market. Other web servers such as Sun Java System Web Server and Zeus occupy a very small share of the market, as shown in the Table 1 above. Since the total share of all of SJSWS and Zeus added together represents less than 10% of the market share, very few vulnerabilities have been found in them and hence the data for these servers has not been used in our study.
(4)
where λvu is a parameter analogous to failure intensity in SRGMs and B is another parameter. B represents the number of vulnerabilities that will eventually be found. We will refer to the model given by Equation 4 as the Effort-Based Model.
Even though Apache and IIS are the top web servers, there is a marked gap between the Apache and IIS market shares, as shown in Figure 2. This difference in market share may be due to several factors. Perhaps the most important of these is that Apache is available for all major operating system platforms and can be obtained without cost. Apache may also have benefited from not having been exposed to serious security issues such as the Code Red [20] or Nimda worms that were faced by IIS in 2001.
3. MODELING VULNERABILTIES IN HTTP SERVERS In this section, the datasets for the total vulnerabilities of the Apache and Microsoft IIS web servers are fitted to the models. The goodness of fit is evaluated to determine how well the models reflect the actual vulnerabilities discovery process. The vulnerabilities data are from the National Vulnerabilities Database maintained by NIST. The market share data from Netcraft [23] was used. We note that Apache represents an open source software and IIS represents a closed source, i.e., a
3
means that vulnerabilities discovery for Apache can be expected to continue at a significant pace in near future
80 Apache
70
IIS
Figure 3 (b) shows cumulative vulnerabilities by number of Apache installations in terms of million system-months and the fitted effort-based model. This effort-based model shows that Apache has not yet approached the saturation phase since the number of vulnerabilities continues to increase approximately linearly as the number of Apache severs increases. The results of the analysis are given in Table 2.
60 50 %
40 30 20
100
10
90 80
0
Vulnerabilities
Fe b00 Ju n00 O ct00 Fe b01 Ju n01 O ct01 Fe b02 Ju n02 O ct02 Fe b03 Ju n03 O ct03 Fe b04 Ju n04 O ct04 Fe b05 Ju n05 O ct05
70
(a) Percentage of Market Share 80000000 Apache
IIS
60 50 40
Total # of System
30
70000000
20 10
50000000
0 M ar -9 Se 6 p9 M 6 ar -9 7 Se p9 M 7 ar -9 Se 8 p9 M 8 ar -9 Se 9 p9 M 9 ar -0 Se 0 p0 M 0 ar -0 Se 1 p0 M 1 ar -0 Se 2 p0 M 2 ar -0 Se 3 p0 M 3 ar -0 Se 4 p0 M 4 ar -0 Se 5 p05
Number of System
60000000
40000000
(a) Time-Based Model
30000000 100
20000000
90
10000000 80
0
Vulnerabilities
Fe b0 Ju 0 n00 O ct0 Fe 0 b0 Ju 1 n0 O 1 ct01 Fe b0 Ju 2 n02 Oc t-0 2 Fe b0 Ju 3 n03 O ct03 Fe b0 Ju 4 n04 O ct0 Fe 4 b0 Ju 5 n05 O ct05
70
(b) Number of Web Servers Figure 2. Server Market Share Trends
3.2 Modeling Apache Vulnerabilities
60 50 40 30
The Apache HTTP server was first released in middle of 1995. Since then it has gained wide popularity and is used by over 50 million web server systems. In this section, we fit the vulnerabilities data for Apache to the time-based and the effortbased models. Figure 3 gives the vulnerabilities data from NVD for the period between March 1996 and December 2005, and the Netcraft market share data coves the period from March 1996 to December 2005.
20 10 0 0
150
300
450
600
750
900
1050
1200
1350
1500
1650
1800
1950
Million System-Months
(b) Effort-Based Model Figure 3. Fitting Apache Vulnerabilities Data
In Figure 3, the bold black lines indicate the fitted models, while the other lines show cumulative vulnerabilities for Apache. Figure 3 (a) shows cumulative vulnerabilities by month for the time-based model. At the beginning, the slope of the curve for Apache rises gently until about January 2000, after which the slope has remained steady. From the point of the three phases of the vulnerabilities discovery process [1], Apache has not yet entered the saturation phase; one or two vulnerabilities are still found each month. Apache currently appears to be in the linear phase, since the number of vulnerabilities still appears to be growing linearly. Despite having been on the market for several years, Apache has not reached the saturation phase possibly because of its larger market share; moreover, the number of systems using the Apache web server is still increasing. This
3.3 Modeling IIS Vulnerabilities Microsoft IIS was released in the early part of 1996. IIS is a popular commercial web server with about 15 million installations currently. In this section, we fit the IIS data to the time-based and the effort-based models. We have used the vulnerabilities data from January 1997 to December 2005. Figure 4 (a) shows the cumulative vulnerabilities by month and the fitted time-based model for the IIS web server. The timebased and effort-based models fit the data for IIS very well. The IIS web server appears to have reached the saturation phase. In recent months, the vulnerabilities discovery rate for IIS has dropped to a very low point. A possible explanation for this can
4
Table 2. χ2 Goodness of Fit Test Results for Total Number of Vulnerabilities Model
Time-Based Model
Effort-Based Model
A
B
C
χ
χ critical
P-value
B
λVU
χ2
χ2 critical
P-value
Apache
.00059
94.05
.8357
45.23
144.35
1
120
.0008
24.267
54.57
.959
IIS
.00074
119.9
.5956
40.3
133.25
1
114.6
.0104
41.763
164.21
.999
Win 98
.0048
37.73
0.554
7.365
60.481
1
37
.0005
3.510
44.9853
1
Win NT4
.0006
136
0.522
35.58
103.01
1
108
.0030
15.05
42.5569
0.985
Software
2
2
be that the number of IIS web servers installed appears to be stationary, unlike the Apache server which is still gaining in terms of new installations. Another possibility is that the number of remaining undiscovered vulnerabilities may actually have dropped significantly.
3.4 Chi-Square Analysis of Goodness of Fit for Apache and IIS In this sub-section, we examine the fit of the models to the data as shown in Figures 3 and 4 above. For χ2 goodness of fit test, we chose an alpha level of 5%. Table 2 gives the chi-square values and parameter values for both the time-based and effortbased models. For comparison, we also provide corresponding parameter values for the Windows 98 and NT operating systems, as well as the chi-square values.
Figure 4 (b) shows cumulative vulnerabilities for the IIS server and the effort-based model by million system-months. Unlike Figure 3 (b), Figure 4 (b) shows a significant degree of saturation.
Table 2 shows that the chi-square values are less than the critical values. This demonstrates that the fit for Apache, IIS, Windows 98 and NT is significant. The fit was obtained by minimizing the χ2 value. Both data sets fit both models with χ2 P-values ranging from 0.959 to nearly 1, indicating that the fit is quite significant. We can also note that parameter A is always less than 0.005 and parameter C is always less then 0.85, while parameter B corresponds approximately to the number of vulnerabilities.
140
120
Vulnerabilities
100
80
60
4. MODELLING VULNERABILITIES CATAGORIES
40
In the previous section we examined the application of the timebased and the effort-based model for the total number of vulnerabilities of Apache and IIS. In this and the following section, we apply these models to two separate classification schemes for server vulnerabilities.
Oct-05
May-05
Jul-04
Dec-04
Feb-04
Sep-03
Apr-03
Nov-02
Jan-02
Jun-02
Aug-01
Oct-00
Mar-01
May-00
Jul-99
Dec-99
Feb-99
Sep-98
Apr-98
Nov-97
Jan-97
0
Jun-97
20
(a) Time-Based Model
Distinguishing among vulnerabilities is useful when we want to examine the nature and extent of the problem. It can help determine what protective actions would be most effective. Vulnerabilities taxonomy is still an evolving area of research. Several taxonomies have been proposed [7], [9], [15], [29]. An ideal taxonomy should have such desirable properties as mutual exclusiveness, clear and unique definition, and coverage of all software vulnerabilities.
140
120
100
80
Vulnerabilities can be classified using schemes based on cause, severity, impact and source, etc. In this analysis, we use the classification scheme employed by the National Vulnerability Database of the National Institute of Standards and Technology. This classification is based on the causes of vulnerabilities. The eight classes are as follows [22], [28]:
60
40
20
1.
0 5
20 35 50 65 80 95 120 150 180 210 240 270 300 330 360 390 420 450 480 510 540 570 600 630 660
Million System-Months
(b) Effort-Based Model
2.
Figure 4. Fitting IIS Vulnerabilities Data
5
Input Validation Error (Boundary condition error, Buffer overflow): Such types of vulnerabilities include failure to verify the incorrect input and read/write involving an invalid memory address. Access Validation Error: These vulnerabilities cause failure in enforcing the correct privilege for a user.
Table 3. Vulnerabilities Classified by Category
3.
Exceptional Condition Error: These vulnerabilities arise due to failures in responding to unexpected data or conditions. 4. Environmental Error: These vulnerabilities are triggered by specific conditions of the computational environment. 5. Configuration Error: These vulnerabilities result from improper system settings. 6. Race Condition Error: These are caused by the improper serialization of the sequences of processes. 7. Design Error: These are caused by improper design of the software structure. 8. Others: Includes vulnerabilities that do not belong to the types listed above, sometimes referred to as nonstandard. Unfortunately, the eight classes are not completely mutually exclusive. Table 3 shows how vulnerabilities are distributed among categories for both the datasets studied. The number of input validation errors is much higher than other types of vulnerabilities for both Apache and IIS. A similar distribution is observed in both operating systems, with input validation errors forming the largest category.
40
Input Validation Error Exceptional Condition Error
35
Vulnerabilities
30 25 20 15 10 5
May-
Oct-05
Jul-04
Dec-04
Sep-03
Feb-04
Apr-03
1500
Nov-02
1400
Jan-02
Jun-02
Aug-01
Mar-01
30
26 (19.85%) 22 (20.18%)
20
10
50%
5
Figure 5. Vulnerabilities by Category Figure 5 compares vulnerabilities distributions in Apache and IIS. The categories with the highest proportions are input validation errors, followed by design errors. There is a slight difference in category ordering between Apache and IIS, with Apache having more configuration errors than access validation
Million System-Months
(b) Effort-Based Model Figure 6. Apache Fitting by Category
6
1900
1800
1700
1600
1300
1200
1100
1000
900
800
0
0 400
40%
300
20% 30% Percentage
25
15
59 (45.04%) 41 (37.61%) 10%
Exception Condition Error
35
15 (11.45%) 18 (16.51%)
Input Validation Error 0%
IIS Apache
16 (12.21%) 6 (5.50%)
Design Error
May-
Desing Error
6 (4.58%) 12 (11.01%)
Exceptional C ondition Error
Input Validation Error
40
4 (3.05%) 4 (3.67%)
200
Access Validation Error
Oct-00
(a) Time-Based Model 45
1 (0.76%) 2 (1.83%)
C onfiguration Error
Jul-99
Mar-96
0
100
C a te g o ry
Design Error
4 (3.05%) 4 (3.67%)
Enviromental Error
4 (3.67%) 4 (3.05%) 3 (1.19%) 0 (0%)
45
Vulnerabilities
Race C ondition Error
Other
When we compare HTTP servers and operating systems, we find a comparable pattern. However, we note that operating systems have a slightly higher proportion of input validation errors and fewer configuration error vulnerabilities. Otherwise, they are within close range of each other.
Because a vulnerability can belong to more than one category, the summation of all categories for a single software system may add up to more than the total number of vulnerabilities (also the percentages may exceed 100%). This is shown in Table 3. Other
Race Condition Error 2 (1.83%) 1 (0.76%) 1 (0.40%) 3 (1.88%)
errors; however, IIS has more access validation errors. While IIS has been more vulnerable to access validation errors, the fact that Apache has been more vulnerable to configuration errors may be due to Apache’s more complex installation requirements.
Dec-99
160
Feb-99
Win XP
700
252
Sep-98
Win 2000
Environmental Error 4 (3.67%) 4 (3.05%) 5 (1.59%) 2 (1.25%)
600
121
Apr-98
IIS
22 (20.18%) 26 (19.85%) 67 (26.59%) 30 (18.75%)
Configuration Error 12 (11.01%) 6 (4.58%) 9 (3.97%) 0 (0%)
500
94
Access Validation Error 6 (5.50%) 16 (12.21%) 20 (7.94%) 10 (6.25%)
Nov-97
Apache
Exceptional Condition Error 18 (16.51%) 15 (11.45%) 45 (17.86%) 27 (16.88%)
Jan-97
Software
Design Error
Jun-97
Input Validation Error 41 (37.61%) 59 (45.04%) 113 (44.84%) 88 (55%)
Total
Aug-96
Type
Table 4. Apache and IIS’s Category Chi-Square Analysis of Goodness of Fit Time-Based Model (Apache)
Parameter Type
Apache
Input Validation Error Design Error Exceptional Condition Error Input Validation Error Design Error Access Validation Error
IIS
B
C
χ2
χ 2 Critical P-Value
B
λVU
χ2
.00138
20
100
38.05
144.35
1
45.72
.0011263
13.426
54.57
.999
.00495
41.641
3.005
38.464
144.35
1
51.45
.0003304
14.77
54.57
.999
.00248
18.004
1.76
53.273
144.35
1
34.78
.0003728
19.30
54.57
.994
.00129
59
.8932
15.55
133.25
1
59
.0057
2.085
24.99
.999
.00261
25
1
34.25
133.25
1
25
.0077
14.77
24.99
.997
.01
14
10
17.29
133.25
1
14
.0604
7.8E-5
24.99
1
Figure 6 shows the time-based model (a) and the effort-based model (b) fitting of Apache by category. In Figure 6, we only consider the three major categories, examining only: input validation errors, design errors and exceptional handling condition errors. 70
Table 4 shows the chi-square goodness of fit tests for the Apache and IIS models by category. Table 4 demonstrates that the chi-square values for each category are less than the critical values. Since χ2 < χ2Critical and since the P-values are close to 1, the fit of input validation, design and exceptional condition error classes are significant for both models.
Design Error Input Validation Access Validation Error
Vulnerabilities
50
40
5. MODELLING VULNERABILITIES BY SEVERITY
30
20
Severity is another way to classify vulnerabilities. The severity of a vulnerability indicates how serious the impact of an exploitation can be. Severity is usually subdivided into three categories, high, medium and low. Recently, NVD used CVSS metric for vulnerability severity with ranges from 1 to 10; CVSS uses many factors to determine the severity. where the range 13.99 corresponds to low severity, 4-6.99 to medium severity and 7-10 to high severity; The National Vulnerability Database of the National Institute of Standards and Technology describes the severity levels, as follows [22]:
Oct-05
May-05
Jul-04
Dec-04
Feb-04
Sep-03
Apr-03
Nov-02
Jan-02
Jun-02
Aug-01
Oct-00
Mar-01
May-00
Jul-99
Dec-99
Feb-99
Sep-98
Apr-98
Nov-97
Jan-97
Jun-97
10
0
(a) Time-Based Model 70
Input Validation
60
Design Error Access Validation Error
1.
50
Vulnerabilities
χ 2 critical P-value
Figure 7 shows the time-based model (a) and effort-based model (b) fitting of IIS vulnerabilities by category. As we mentioned above, the IIS model has a better fit than the Apache model, since IIS has reached the saturation phase. The categorized number of vulnerabilities shows the same pattern as demonstrated by the total number of vulnerabilities. Thus, each category shows a related pattern with regard to total number of vulnerabilities. Our time-based and effort-based models are fitted for each category. It may be noted that the number of input validation errors and design errors are the most common category in Apache and IIS.
We plot the vulnerabilities for the major categories to determine whether there is an observable pattern at the level of individual classes. Since we noted a similar pattern for the uncategorized vulnerabilities, a possible fit was examined. Figures 6 and 7 show the fit for the Apache and Microsoft IIS, respectively.
60
Effort-Based Model (Apache)
A
40
30
2.
20
10
3.
0
0
50
100 150 200 250 300 350 400 450 500 550 600 650 700 Million System-Months
(b) Effort-Based Model Figure 7. Fitting IIS by Category
7
High Severity: This makes it possible for a remote attacker to violate the security protection of a system (i.e., gain some sort of user, root or application account), or permits a local attack that gains complete control of a system, or if it is important enough to have an associated CERT/CC advisory or US-CERT alert. Medium Severity: This does not meet the definition of either “high” or “low” severity. Low Severity: The vulnerability typically does not yield valuable information or control over a system but rather gives the attacker knowledge provides the attacker with information that may help him find and exploit other vulnerabilities or we feel that the vulnerability is inconsequential for most organizations
Table 5. Apahce and IIS’s Severity Chi-Square Analysis of Goodness of Fit Parameter
Time-Based Model (Apache)
Severity
Effort-Based Model (Apache)
A
B
C
χ2
χ 2 Critical
P-Value
B
λVU
χ2
χ 2 critical
P-value
Apache
High
.00156
27.00
1.00
42.1
144.3
1
32
.00095
19.13
54.57
1
Medium
.00495
41.64
3.00
55.8
144.3
.999
10
.0001
23
54.57
1
Low
.00248
18.00
1.76
15.7
144.3
1
57
.00014
10.18
54.57
1
High
.00176
38
.999
28.2
133.2
1
38
.0006
2.8
23.68
1
Low
.00127
77.9
1.21
53.5
133.2
.999
74
.0112
5.4
23.68
1
IIS
The distributions of the severities of the Apache and IIS vulnerabilities show similarity. About 60% of total vulnerabilities have low severity, followed by about 30% with high severity, with medium severity vulnerabilities at about 4 to 10%. This shows that while low severity vulnerabilities, i.e., those that do not cause serious impact, are the majority, the fraction of high severity vulnerabilities is nevertheless substantial and represents a significant threat to the server.
Surprisingly, both Apache and IIS show a similar pattern. A large fraction of the high severity vulnerabilities is found early, while the discovery of low severity vulnerabilities is at about 80% after two or three years. Later, high severity vulnerabilities start to form a larger proportion at the expense of low severity vulnerabilities. 60
High
50
100% High Medium
80%
40 Vulnerabilities
Low
70% Percentage
Medium Low
90%
60% 50%
30
20
40% 30%
10
20%
1700
Sep-05
Sep-04
1600
Mar-05
Sep-03
Mar-04
1500
Sep-02
Mar-03 1400
Sep-01
Mar-02
Sep-00
Mar-01
Sep-99
Mar-00
Sep-98
Mar-99
Sep-97
Mar-98
Sep-96
Mar-96 May-
Oct-05
Jul-04
Dec-04
Feb-04
Sep-03
Apr-03
Nov-02
Jan-02
Jun-02
Aug-01
Mar-01
May-
Oct-00
Jul-99
Dec-99
Feb-99
Sep-98
Apr-98
Nov-97
Jan-97
Jun-97
Aug-96
Mar-96
0%
Mar-97
0
10%
(a) Time-Based Model
Time
60
(a) Apache 100%
50
High 90%
Mideum Low
Low
40 Vulnerabilities
80% 70% 60%
Percentage
High
Medium
50%
30
20
40%
10
30% 20%
1900
1800
1300
1200
1100
900
1000
800
700
600
500
400
300
200
0
100
0
10%
Million System-Months Oct-05
Dec-04
May-05
Jul-04
Feb-04
Sep-03
Apr-03
Nov-02
Jun-02
Jan-02
Aug-01
Oct-00
Mar-01
May-00
Dec-99
Jul-99
Feb-99
Sep-98
Apr-98
Nov-97
Jan-97
Jun-97
0%
(b) Effort-Based Model
Time
Figure 9. The Fit of Apache’s Severity
(b) IIS
We apply the time-based and effort-based models to the three Apache severity classes. In Figures 9 and 10, the bold lines indicate the fitted time-based and effort-based models for each category. Figure 9 shows the result of fitting the time-based (a) and the effort-based model (b) to the three severity classes.
Figure 8. Percentage of Cumulative Vulnerabilities Categorized by Severity Figure 8 plots the percentage of the cumulative number of vulnerabilities for each severity class for each month.
8
Figure 10 shows the fit for time-based (a) and effort-based model (b) for IIS severity classes. In severity classes, the IIS vulnerabilities data had attained the saturation phase; while the Apache’s vulnerabilities are still being discovered.
100% 90% 80%
Table 5 shows chi-square analysis of goodness of fit for the Apache and IIS by severity level. Using regression analysis, we obtained parameter values from Figures 9 and 10 above. As was done previously, for chi-square goodness of fit test, we chose an alpha level of 5%. The chi-square and parameter values for time-based and effort-based models are also shown in Table 5. This chi-square test shows that the fit for the three severity categories is significant, and the chi-square test shows that the vulnerabilities classified by severity datasets fit the model.
P ercen tag e
70% 60% 50% 40% 30% 20% 10% 0% High
90
Input Validation Error Access Validation Error Race Condition
80 High
Design Error Configuration Error Other
Low
Exceptional Condition Error Environmental Error
Figure 11. Apache Severity by Category
Low
70
Medium Severity
Vulnerabilities
60 100%
50
90%
40
80% 70% Percen tag e
30 20 10
60% 50% 40% 30% 20%
Oct-05
May-05
Jul-04
Dec-04
Feb-04
Sep-03
Apr-03
Nov-02
Jan-02
Jun-02
Aug-01
Oct-00
Mar-01
May-00
Jul-99
Dec-99
Feb-99
Sep-98
Apr-98
Jun-97
Nov-97
Jan-97
0
10% 0%
(a) Time-Based Model
High
Medium
Low
Severity
90 80
Input Validation Error Access Validation Error Race Condition
High Low
Exceptional Condition Error Environmental Error
Figure 12. IIS Severity by Category
70
Vulnerabilities
Design Error Configuration Error Other
60
6. DISCUSSION
50
When the total number of vulnerabilities is examined, both the time-based and effort-based models fit the datasets well, even when the vulnerabilities are categorized by type or severity. This suggests that the models can be used to estimate the number of vulnerabilities expected to be discovered in a given period, and which types and severity level is likely to dominate.
40 30 20 10
The results of model fitting for the vulnerabilities classified by type are shown in Table 4. The fitting was done for the most common types of vulnerabilities for which the available data is statistically significant. It would be difficult to use these models to estimate the types of vulnerabilities that occur less frequently because the data may not be sufficiently statistically significant to make meaningful projections.
0 0
50
100
150
200
250
300
350
400
450
500
550
600
650
700
Million System-Months
(b) Effort-Based Model Figure 10. The Fit of IIS’s Severity Figures 11 and 12 illustrate how severity level correlates with error classification. It is noticeable that the input validation error constituted the majority among high severity vulnerabilities for both Apache and IIS. In Apache, a relatively smaller fraction of exception condition errors are of high severity. In IIS as well, the exception condition errors tend to be from among the vulnerabilities with low severity. For IIS, most configuration errors are medium severity.
The results of model fitting for the vulnerabilities classified by severity are shown in Table 5. In all cases, there is enough data for the high and low severity vulnerabilities, and the fit is quite good. The results suggest that these two models can be used to project the expected number of high severity vulnerabilities. The effort-based model requires the use of the market share data, which may be difficult to obtain. The time-based model does not require this data; it can therefore be a feasible alternative when
9
market share data is unavailable. Further research needs to be done to evaluate the predictive capabilities of the two models.
remaining vulnerabilities, economic aspects etc., also need to be considered when choosing a web server.
Analysis of the vulnerabilities classified by severity using the National Vulnerability Database standards for severity classification shows that a large fraction of the vulnerabilities initially found are high risk. However, subsequently a larger fraction of low severity vulnerabilities are encountered within the first and second years. Later, there is again a slight rise in the fraction of high severity vulnerabilities found. This variability was observed for both servers. Further research is needed to identify the cause of this variability.
7. CONCLUSIONS This paper explores the applicability of quantitative models for the number of vulnerabilities and vulnerabilities discovery rates for HTTP servers. This study has demonstrated that the vulnerabilities discovery process in servers follows a pattern, which can be modeled. It is therefore possible to make reasonable projections about the number of remaining vulnerabilities and vulnerabilities discovery rates. We also examined the application of the models to vulnerabilities belonging to specific categories. The fit was significant for both the time-based and the effort-based models. The distribution of the vulnerabilities into specific categories was also analyzed and compared with distributions in the operating systems. The results show that the distributions are comparable for the distributions of the operating systems.
Static analysis has been used in software reliability engineering, where some of the systems’ attributes are estimated empirically even before testing begins. Similar static analysis can be carried out by utilizing metrics such as software size and estimated number of total defects. These methods can potentially be used to estimate Defect density (DKD) and Vulnerability density (VKD), which can then be used to estimate the total number of vulnerabilities of a comparable system. DKD gives the defects per thousand lines of code and VKD is the number of vulnerabilities per thousand lines of code. Table 6 below shows some of the major attributes of the Apache server and two other major operating systems for comparison. Unfortunately, some of the important metrics for the Microsoft IIS server were not available to us at the time this paper was written. For proprietary systems, such data can be hard to obtain outside of the developing organization.
Vulnerabilities categorized by severity were also plotted to determine whether there is a correlation between vulnerabilities type and severity level. It was observed that a larger number of input validation error vulnerabilities constitute a high risk. This suggests that more effort should be spent on testing in order to target vulnerabilities from this class, thereby minimizing the number of high risk vulnerabilities. The results indicate that the models originally proposed for operating systems are also applicable to servers. These models can be used to estimate vulnerabilities discovery rates, which can be integrated with risk assessment models in the future. A model recently proposed by Sahinoglu [26] needs such an assessment for estimating risk and cost of loss. Furthermore, these models can be integrated into the development process to create more secure software systems [30].
Table 6. Known Defect Density vs. Known Vulnerability Density Application
SLOC
Known Defects
DKD
Known Vulnerabilities
VKD
Ratio VKD/DKD
Apache
227,410
2212
0.972
94
0.0413
.04252
IIS
N/A
N/A
N/A
121
N/A
N/A
6.25
81
0.0005
.0081
5.56
179
0.001
.0179
Windows 16,000,000 10,000 98 Windows 18,000,000 10,000 NT 4.0
Further work is needed to evaluate the prediction accuracy of the models so that the users can measure how accurately these models can predict future vulnerabilities discovery rates. Further research is also needed to evaluate the degree of confidence that can be attained when these methods are used to predict the type of vulnerabilities that are anticipated and their severity levels.
REFERENCES
It is likely that the sizes of IIS and Apache are comparable in terms of SLOC numbers, since both offer the same features. In Table 6, we observe that vulnerability density values for the Windows systems are significantly less than for Apache. This may be due to the fact that Windows software has large segments that do not play a role in accessibility, while severs are smaller and therefore vulnerabilities are more concentrated in the code. This assumption is supported by the fact that the defect density to vulnerability density ratio is higher in Windows NT 4.0, a server operating system, than in Windows 98.
[1] Alhazmi, O. H., and Malaiya, Y. K. Quantitative vulnerability assessment of system software. Proc. Annual Reliability and Maintainability Symposium (Jan. 2005), 615–620. [2] Alhazmi, O. H., Malaiya, Y. K., and Ray, I. Security vulnerabilities in software systems: A quantitative perspective. Proc. Ann. IFIP WG11.3 Working Conference on Data and Information Security (Aug. 2005), 281–294. [3] Alhazmi, O. H., and Malaiya, Y. K. Modeling the vulnerability discovery process. Proc. 16th International Symposium on Software Reliability Engineering (Nov. 2005), 129–138.
After comparing the vulnerabilities trends of the web servers discussed in this paper, it is expected that fewer vulnerabilities will be discovered in IIS in the future. This may lead to the conclusion that IIS is more secure than Apache in this respect. However, this is simply due to the fact that IIS has reached saturation phase, even though more IIS vulnerabilities have been found in the past. Other factors such as patch release, number of
[4] Alhazmi, O. H., and Malaiya, Y. K. Prediction capability of vulnerability discovery process. Proc.Reliability and Maintainability Symposium (Jan. 2006).
10
[5] Anderson, R. Security in open versus closed systems—the dance of boltzmann, coase and moore. In Conf. on Open Source Software: Economics, Law and Policy (2002), 1–15.
[17]
[6] Apache Software Foundation Bug System, http://issues.apache.org/bugzilla/. [7] Aslam, T., and Spafford E. H. A taxonomy of security faults. Technical report, Carnegie Mellon, 1996.
[18]
[8] Aura, T., Bishop, M., and Sniegowski, D. "Analyzing Single-Server Network Inhibition," Proceedings of the 13th IEEE Computer Security Foundations Workshop pp.108117 (July 2000)
[19] [20]
[9] Bishop, M. Vulnerability analysis: An extended abstract. Proc. Second International Symposium on Recent Advances in Intrusion Detection (Sept. 1999), 125-136.
[21]
[10] Brocklehurst, S., Littlewood, B., Olovsson T. and Jonsson, E. On measurement of operational security. Proc. 9th Annual IEEE Conference on Computer Assurance (1994), 257–266.
[22] [23] [24]
[11] Browne, H. K., Arbaugh, W. A., McHugh, J., and Fithen, W. L. A trend analysis of exploitations. In IEEE Symposium on Security and Privacy (2001), 214–229. [12] Ford, R., Thompson, H., and Casteran, F. Role comparison report—web server role. Technical Report, Security Innovation, 2005. [13] Hallberg, J., Hanstad, A., and Peterson, M. A framework for system security assessment. Proc. 2001 IEEE Symposium on Security and Privacy (May 2001), 214–229. [14] Kargl, F., Maier, J., and Weber, M. Protecting web servers from distributed denial of service attacks. Proc. 10th International WWW Conference (2001), 514–524. [15] Landwehr, C. E., Bull, A. R., McDermott, J. P., and Choi, W. S. A taxonomy of computer program security flaws. ACM Comput. Surv. 26, 3 (1994), 211–254. [16] Littlewood, B., Brocklehurst, S., Fenton, N. E., Mellor, P., Page, S., Wright, D., Dobson, J., McDermid, J., and Gollmann, D. Towards operational measures of computer
[25] [26]
[27]
[28] [29]
[30]
11
security. Journal of Computer Security 2, 2-3 (1993), 211– 230. Lyu, M. R. Handbook of Software Reliability. McGrawHill, 1995. Madan, B. B., Goseva-Popstojanova, K., Vaidyanathan, K., and Trivedi, K. S. A method for modeling and quantifying the security attributes of intrusion tolerant systems. Perform. Eval. 56, 1-4 (2004), 167–186. Mitre Corp, Common Vulnerabilities and Exposures, http://www.cve.mitre.org/. Moore, D., Shannon, C., and Claffy, K. C. Code-red: a case study on the spread and victims of an internet worm. In Internet Measurement Workshop (2002), pp. 273–284. Musa, J. Software Reliability Engineering. McGraw-Hill, 1999. National Vulnerability Database . http://nvd.nist.gov/. Netcraft,. http://news.netcraft.com/. Rescorla, E. Security holes... who cares? Proc. 12th USENIX Security Symposium (2003). Rescorla, E. Is finding security holes a good idea? IEEE Security and Privacy 03, 1 (2005), 14–19. Sahinoglu, M. Quantitative risk assessment for dependent vulnerabilities. Proc. Reliability and Maintainability Symposium (Jan. 2006), 82-85. Schultz, E. E., Brown, D. S., and Longstaff, L. T. A. Responding to computer security incidents. Lawrence Livemore National Laboratory (July 1990). Securityfocus, http://www.securityfocus.com/. Seacord, C. R. and Householder, A. D. A structured approach to classifying vulnerabilities. Technical Report CMU/SEI-2005-TN-003, Carnegie Mellon, 2005. Seacord, R. Secure Coding in C and C++. Addison Wisely, 2005.
