Contributions for Improving Genetic Programming-Based Approaches to the Evolutionary Testing of Object-Oriented Software by José Carlos Bregieiro Ribeiro Supervisors: F. Fernández de Vega Mário Zenha-Rela Mérida, Spain 12th November, 2010
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
1 / 68
Outline 1 Introduction
Background Related Work 2 Methodology Overview
The eCrash Framework 3 New Techniques
Fitness Evaluation Strategy Input Domain Reduction Technique Adaptive Evolutionary Testing Approach Object Reuse Methodology 4 Conclusions
Future Work 5 Tool Demonstration José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
2 / 68
Introduction
Outline 1 Introduction
Background Related Work 2 Methodology Overview
The eCrash Framework 3 New Techniques
Fitness Evaluation Strategy Input Domain Reduction Technique Adaptive Evolutionary Testing Approach Object Reuse Methodology 4 Conclusions
Future Work 5 Tool Demonstration José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
3 / 68
Introduction
Introduction
Software Testing is the process of exercising an application to detect errors and to verify that it satisfies its requirements. • It is a time consuming, difficult and expensive process. •
Evolutionary Testing Automating the Test Data generation process is vital to advance the state-of-the-art in Software Testing. • The application of Evolutionary Algorithms to Test Data Generation is often referred to as Evolutionary Testing. •
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
4 / 68
Introduction
Introduction •
Evolutionary Testing is an emerging methodology for generating Test Data for Object-Oriented Software.
•
It is, however, a difficult subject, especially if the aim is to • • •
implement an automated solution, viable with a reasonable amount of computational effort, which is adaptable to a wide range of test objects.
Our Approach •
This Thesis presents a Genetic Programming-based approach to the generation of structural Unit Test data for Object-Oriented software.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
5 / 68
Introduction
Background
Software Testing • Testing is inherent to every phase of Software
development.
Unit Testing • Unit Testing verifies individual application objects
or methods in an isolated environment.
Why is Unit Testing so important? • Most errors are introduced at the Coding Stage. • It enables the detection and removal of errors at a
much smaller cost. • Unit Testing is largely Structure-oriented. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
6 / 68
Introduction
Background
Object-Orientation • Most work in Testing has been done with “procedure-oriented”
software in mind. • The Object-Oriented paradigm poses several hindrances to testing
due to some aspects of its very nature.
The State Problem • Due to the Encapsulation principle of the OO paradigm: • the only way to change the state of an object is through the execution of a series of method calls; • the only way to observe the state of an object is through its operations. •
The state of an object is accessible only through an interface of public methods.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
7 / 68
Introduction
Background
Object Orientation • A Unit Test for Object-Oriented programs usually requires: • at least, an instance of the Class Under Test (CUT); • additional objects, which are required (as parameters) for the instantiation of the CUT and for the invocation of the Method Under Test (MUT) – and for the creation of these additional objects, more objects may be required; • putting the participating objects into particular states, in order for the test scenario to be processed in the desired way – and, consequently, method calls must be issued for these objects.
Example Test Case
José Carlos Bregieiro Ribeiro (IPL)
• Method Under Test:
• Invocation of on aims at changing the state of before passing it to . • Test Cluster:
Ph.D. Thesis Presentation
12th November, 2010
8 / 68
Introduction
Background
Evolutionary Algorithms Evolutionary Algorithms •
Evolutionary Algorithms use simulated evolution as a search strategy to evolve candidate solutions for a given problem, using operators inspired by genetics and natural selection.
• The best known algorithms in this
class include: • • • • José Carlos Bregieiro Ribeiro (IPL)
Evolution Strategies, Evolutionary Programming, Genetic Algorithms, and Genetic Programming.
Ph.D. Thesis Presentation
12th November, 2010
9 / 68
Introduction
Background
Evolutionary Testing
•
The goal of Evolutionary Testing problems is to find a set of Test Cases that satisfies a certain test criterion.
The test objective must be defined numerically – i.e., suitable fitness functions, that provide guidance to the search by telling how good each candidate solution is, must be defined. The search space is the set of possible inputs to the Test Object. •
Defining a Test Set that achieves full structural coverage may involve the generation of complex and intricate Test Programs in order to define elaborate state scenarios.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
10 / 68
Introduction
Background
Object-Oriented Evolutionary Testing • Object-Oriented Evolutionary Testing requires the definition of a
suitable representation for Test Programs.
Genetic Programming emerges as a natural candidate because: GP involves the evolution of tree structures and Test Programs may be represented as Method Call Trees (MCTs). The typing mechanisms available in GP facilitate the encoding of Object-Oriented programs. A GP tree can vary in length throughout the run, thus allowing experimenting with different sized Test Programs. GP evolves active structures and the solutions can be executed without post-processing. It is possible to systematically define the Function Set solely with basis on Test Cluster information. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
11 / 68
Introduction
Related Work
Related Work • Evolutionary Algorithms and other metaheuristics have been applied, with significant success, to automate Test Data generation for OO Software. Examples: Genetic Algorithms [Ton04, CKP05, DJAR07, IX07, LRW07, IX08, XTdHS08]; Genetic Programming [SG06, See06, WW06a, WW06b, AY07a, WS07, Wap07, GR08]; Ant Colony Optimization [LWL05]; Memetic Algorithms [AY07b, Arc08]; Estimation of Distribution Algorithms [SAY07]; Artificial Immune Systems [LR08]. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
12 / 68
Introduction
Related Work
Related Work – Chronology
Miller and Spooner [MS76] are typically considered the first to combine the results of actual executions of a program with a search technique; Xanthakis et al. [XES+92] were the first to apply Evolutionary Algorithms to generate structural Test Data; Tonella [Ton04] was the first to propose an Object-Oriented Evolutionary Testing technique, for generating input sequences for the structural Unit Testing of Java classes by means of GAs. Wappler et al. [WW06b, WW06a], and Seesing et al. [SG06, See06] presented the first GP-based approaches to Object-Oriented Evolutionary Testing in 2006.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
13 / 68
Introduction
Related Work
Related Work – Observations
• Object-Oriented Evolutionary Testing is a fairly recent field of
study, and is yet to be investigated comprehensively. The program’s structure is the basis for nearly all studies, with the objective being that of attaining a coverage criterion; Java is typically the programming language of choice for the purpose of implementation; Container classes (e.g., Stack, BitSet, Vector, TreeMap) are employed by almost all works for experimentation purposes; The frameworks developed are seldom freely available for other researchers to experiment and compare their approaches. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
14 / 68
Methodology Overview
Outline 1 Introduction
Background Related Work 2 Methodology Overview
The eCrash Framework 3 New Techniques
Fitness Evaluation Strategy Input Domain Reduction Technique Adaptive Evolutionary Testing Approach Object Reuse Methodology 4 Conclusions
Future Work 5 Tool Demonstration José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
15 / 68
Methodology Overview
Approach
• Test Data quality evaluation includes
instrumenting the Test Object, executing it with the generated Test Cases, and tracing the structures traversed in order to derive coverage metrics. • The primary goal is finding a set of
Test Cases that achieves full structural coverage of the Test Object. •
Test Programs are encoded and evolved as STGP individuals.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
16 / 68
2
3
4
5
6
2
3
4
5
6
Figure: Method Call Tree.
Listing: Method Call Sequence.
Listing: Test Program.
Methodology Overview
The eCrash Framework
The eCrash Framework • eCrash is a Java-based prototype Test Data generation tool. • It was developed during the course of the research presented in this PhD Thesis to support the integration of research steps.
Special attention was put on: improving the level of performance and automation of the Software Testing process; minimising the interference of the tool’s users on the Test Object analysis to a minimum; mitigating the impact of users’ decisions in the Test Data generation process.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
18 / 68
Methodology Overview
The eCrash Framework
The eCrash Framework eCrash Modules • Test Object Instrumentation module: builds the CFG, and instruments the Test Object.
• Automatic Test Object Analyser module: defines the Test Cluster, generates the Function Set, and parametrises the Test Data generation process.
• Test Program Generation module: iteratively evolves potential solutions to the problem with basis on the GP paradigm.
• Test Program Eval. and Manag. module: synthesises, executes and evaluates Test Programs dynamically. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
19 / 68
New Techniques
Outline 1 Introduction
Background Related Work 2 Methodology Overview
The eCrash Framework 3 New Techniques
Fitness Evaluation Strategy Input Domain Reduction Technique Adaptive Evolutionary Testing Approach Object Reuse Methodology 4 Conclusions
Future Work 5 Tool Demonstration José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
20 / 68
New Techniques
Fitness Evaluation Strategy
A Strategy for Evaluating Test Programs for the Evolutionary Testing of Object-Oriented Software
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
21 / 68
New Techniques
Fitness Evaluation Strategy
Introduction • The execution of Test Cases may abort prematurely if a runtime
exception is thrown during execution. • When this happens, it is not possible to trace the structures traversed because the final instruction of the Test Program is not reached.
Feasible Test Cases are effectively executed and terminate with a call to the method under test. Unfeasible Test Cases abort prematurely because a runtime exception is thrown.
Example Unfeasible Test Case ! "" ⇒ #$"% #& " ' "" ( "
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
22 / 68
New Techniques
Fitness Evaluation Strategy
Weight Reevaluation and Fitness Calculation • The issue of steering the search towards the traversal of interesting
CFG paths was addressed by assigning weights the CFG nodes. • Each generation the node’s weight is multiplied by:
the weight decrease constant value α so as to decrease the weight of all CFG nodes indiscriminately; the hit count factor which worsens the weight of recurrently hit nodes; and, the path factor which improves the weight of nodes that lead to interesting nodes and belong to interesting paths.
Weight Reevaluation
José Carlos Bregieiro Ribeiro (IPL)
= (α
)
| |
+½
Ph.D. Thesis Presentation
∈ | |× ¾
12th November, 2010
23 / 68
New Techniques
Fitness Evaluation Strategy
Weight Reevaluation and Fitness Calculation • The fitness of individuals is calculated as follows:
( ) := the average weight of the CFG nodes traversed. ( ) := percentage of instructions executed plus the unfeasible penalty constant β.
Feasible Test Case Evaluation
( ) =
∈ | |
Unfeasible Test Case Evaluation
( ) = β + José Carlos Bregieiro Ribeiro (IPL)
( − )×½¼¼
Ph.D. Thesis Presentation
12th November, 2010
24 / 68
New Techniques
Fitness Evaluation Strategy
Results
•
Our strategy allows unfeasible Test Cases to be considered at certain points of the evolutionary search. •
•
Namely, once the feasible Test Cases being bred cease to be interesting because they only traverse recurrently hit structures.
A good compromise between the intensification and diversification of the search can be achieved. This favours the complexity of Method Call Sequences; • complex Test Programs are often needed to define elaborate state scenarios and achieving full structural coverage. •
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
25 / 68
New Techniques
Input Domain Reduction Technique
Employing Purity Analysis for Reducing the Input Domain of Object-Oriented Evolutionary Testing Problems
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
26 / 68
New Techniques
Input Domain Reduction Technique
Introduction • Object-Oriented Evolutionary Testing problems are hindered by the
explosive size of the search space.
Input Domain Reduction • Input Domain Reduction deals with the removal of irrelevant
variables. • The intention is that of decreasing the number of distinct Test
Programs that can possibly be created while searching for a particular test scenario. • Our approach involves representing Test Programs using STGP. • Purity Analysis provides a means to automatically identify and
remove Function Set entries that do not contribute to the definition of interesting test scenarios. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
27 / 68
New Techniques
Input Domain Reduction Technique
Introduction • A pure method has no externally visible side effects when executed
– i.e., it does not change the state of the objects involved in its call. • A parameter is read-only if the method does not write the parameter or any objects reachable from the parameter; safe if it is read-only, and the method does not create any new externally visible paths in the heap to other objects; read/write otherwise.
Example "
⇒ is safe; method throws #$"% #& " ") ! ! José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
28 / 68
New Techniques
Input Domain Reduction Technique
Purified Function Set Generation
Data: class under test Result: purified function set
test cluster; foreach method in test cluster do foreach parameter do parameter purity; ! types required table; ! provided table; " # EMCDG with nodes; EMCDG nodes with edges; !$ irrelevant edges; purified EMCDG; purified function set;
José Carlos Bregieiro Ribeiro (IPL)
Test Cluster Data Types Members Object Object() Stack Stack() Object pop() Object push(Object) boolean empty() Object peek()
Parameter Purity Analysis Results Method Parameter Purity Object pop() IP is read/write Object push(Object) IP is read/write P0 is read-only boolean empty() IP is safe Object peek() IP is safe int search(Object) IP is safe P0 is safe Ph.D. Thesis Presentation
12th November, 2010
29 / 68
New Techniques
Input Domain Reduction Technique
Purified Function Set Generation
Data: class under test Result: purified function set
test cluster; foreach method in test cluster do foreach parameter do parameter purity; ! types required table; ! provided table; " # EMCDG with nodes; EMCDG nodes with edges; !$ irrelevant edges; purified EMCDG; purified function set;
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
30 / 68
New Techniques
Input Domain Reduction Technique
Purified Function Set Generation
Data: class under test Result: purified function set
test cluster; foreach method in test cluster do foreach parameter do parameter purity; ! types required table; ! provided table; " # EMCDG with nodes; EMCDG nodes with edges; !$ irrelevant edges; purified EMCDG; purified function set;
José Carlos Bregieiro Ribeiro (IPL)
Purified Member Stack() Object pop() Object pop() Object push(Object)
Function Set Return Type Stack [IP] Object [RE] Stack [IP] Object [RE]
Object push(Object)
Stack [IP]
Object peek() Object()
Object [RE] Object [RE]
Entries Excluded Object push(Object) Object [P0] Object peek() boolean empty() int search(Object)
Object [IP] Stack [IP] Stack [IP]
int search(Object)
Stack [P0]
Ph.D. Thesis Presentation
Child Types Stack [IP] Stack [IP] Stack [IP], Object [P0] Stack [IP], Object [P0] Stack [IP]
Stack [IP], Object [P0] Stack [IP] Stack [IP] Stack [IP], Object [P0] Stack [IP], Object [P0]
12th November, 2010
31 / 68
New Techniques
Input Domain Reduction Technique
Results
•
The size of the Input Domain is reduced. •
•
The number of function set entries when purity analysis is used is, on average, 31.6% lower than that obtained when no purity analysis is employed.
The performance of the Test Data generation process is enhanced. For the class, the number of generations required to attain full coverage decreased 66%. • For the class, the percentage of test cases that accomplished full coverage increased approximately 6%. •
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
32 / 68
New Techniques
Adaptive Evolutionary Testing Approach
An Adaptive Approach to the Evolutionary Testing of Object-Oriented Software
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
33 / 68
New Techniques
Adaptive Evolutionary Testing Approach
Introduction • Adaptive Evolutionary Algorithms (AEAs) are distinguished by
their dynamic manipulation of selected parameters or operators during the course of evolving a problem solution.
Advantages of Adaptive Evolutionary Algorithms: more reactive to the unanticipated particulars of a problem; can dynamically acquire information about regularities in the problem and exploit them; allow a bias to be introduced directly into the problem, so as to remove (or penalize) undesirable candidate solutions and improve the efficiency of the search.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
34 / 68
New Techniques
Adaptive Evolutionary Testing Approach
Introduction
• With an increasing size of the Function Set, the probability that the
“right” instructions appear in the Test Programs decreases. • Our strategy for addressing this
problem involves dynamically adapting the Function Set’s constraints selection probabilities. • The introduction of interesting
instructions by means of Mutation is favoured. • The Probabilistic Tree Creation 2
(PTC2) algorithm is utilised for building trees. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
35 / 68
Table: Example Function Set and Type Set. Function Name Return Type void print(Object) TREE String intToStr(Integer) STRING (rank:0.2) “Foo” STRING (rank:0.8) “Bar” STRING (rank:0.2) Integer add(Integer,Integer) INT (rank:0.8) 0 INT (rank:0.4) 1 INT (rank:0.6) Set Types: OBJECT = [STRING, INT]
Child Types OBJECT INT
INT, INT
Figure: Example STGP tree and corresponding Test Program.
Probability of building the STGP tree above 100% as root node × 80% as non-terminal × 40% as terminal × 60% as terminal = 19.2 % probability
New Techniques
Adaptive Evolutionary Testing Approach
Dynamic Adaptation Strategy
Let the Constraint Selection Ranking of constraint in generation be identified as ρ . Then, ρ is updated, at the beginning of each generation, in accordance to the following Equation: ρ = ρ −½ − λ −½ − σ −½ − γ −½ With • λ being the Runtime Exceptions Caused Factor • σ being the Runtime Exceptions Caused by Ancestors Factor • γ being the Constraint Diversity Factor
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
37 / 68
New Techniques
Adaptive Evolutionary Testing Approach
Dynamic Adaptation Strategy ρ = ρ −½ − λ −½ − σ −½ − γ −½ the Runtime Exceptions Caused Factor λ penalises the ranking of constraints corresponding to instructions that have caused runtime exceptions to be thrown in the preceding generation. the Runtime Exceptions Caused by Ancestors Factor σ penalises the ranking of constraints corresponding to instructions which have participated in the composition of sub-trees that have caused runtime exceptions to be thrown in the preceding generation; the higher the ancestry level, the lower the penalty. the Constraint Diversity Factor γ allows constraints to recover their ranking if they have been being used infrequently, and penalises the ranking of constraints which have been selected too often. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
38 / 68
New Techniques
Adaptive Evolutionary Testing Approach
Results • In terms of the percentage of run to attain full structural coverage: • the adaptive strategy outperformed the static approach for 28.4% of the MUTs tested; • whereas the latter only surpassed the former in 5.9% of the situations. • In terms of the average success rate: • the adaptive strategy enhances results by 3% for ; • the improvement is even more significant for the class, with the results meliorating 11%.
• The adaptive strategy also presents the following advantages: • it permits mitigating the negative effects of including a large number
of entries into the Test Cluster; • allows a higher degree of freedom when defining the Test Cluster, by
minimizing the impact of redundant, irrelevant or erroneous choices. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
39 / 68
New Techniques
Object Reuse Methodology
Enabling Object Reuse on Genetic Programming-based Approaches to Object-Oriented Evolutionary Testing
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
40 / 68
New Techniques
Object Reuse Methodology
Introduction
•
If structural adequacy criteria are employed, Object Reuse is a feature of paramount importance.
Object Reuse •
Object Reuse means that one instance can be passed to multiple methods as an argument, or multiple times to the same method as arguments.
•
In the context of OOET, it enables traversal specific structures of software that would not be reachable otherwise.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
41 / 68
2
3
4
2
3
New Techniques
Object Reuse Methodology
Introduction • The method of Java’s class provides a paradigmatic example: for the to return , the same reference must be passed as an argument twice. • Every class has as a superclass; this means that every class inherits , and uses it for equivalence verification.
The ’s example " %&" &" ' ! " %&" Output: false
" %&" &" ' ! " %&" Output: true José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
42 / 68
New Techniques
Object Reuse Methodology
At-Nodes • The proposed approach to Object Reuse involves the definition of
novel type of GP nodes – the At-Nodes.
At-Nodes • At-Nodes are GP nodes that refer to other (standard) GP nodes,
and allow the reuse of portions of the tree. • More specifically, At-Nodes enable the reuse of the object references
returned by the functions corresponding to the reused sub-trees. • This is accomplished by having the node pointed at by the At-Node
provide the parameter not only to its parent node, but also to the At-Node’s parent node. • Parameter assignment is performed during the MCT’s linearisation. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
43 / 68
New Techniques
Object Reuse Methodology
Example with Object Reuse
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
44 / 68
New Techniques
Object Reuse Methodology
The Object Reuse and Reverse Object Reuse Operators • The introduction and removal of At-Nodes is performed by means of
two custom-made evolutionary operators:
Object Reuse Op.
Reverse Object Reuse Op.
• Acts on an individual by selecting two nodes: – the node to be replaced by the At-Node, – and the node to be “pointed at’’ by the At-Node;
• Removes At-Nodes by: – searching the tree for At-Nodes, – and replacing them with copies of the sub-trees pointed at by the At-Nodes.
• and by inserting the newly created At-Node into the tree.
• Allows avoiding the reformulation of Mutation and Crossover.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
45 / 68
New Techniques
Object Reuse Methodology
The Object Reuse and Reverse Object Reuse Operators
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
46 / 68
New Techniques
Object Reuse Methodology
Performance Enhancements The Object Reuse methodology proposed is able to enhance the performance of the Test Data generation process: • it yields solutions with smaller overall size and lower average
structural complexity; • and the feasibility of the generated Test Programs is increased as a
result of the introduction of a specific heuristic for node selection.
Replaced-Destination Node Pair Selection heuristic • Promotes the selection, by the Object Reuse Operator, of GP tree
nodes corresponding to non-executed portions of Test Programs, so as to enhance Test Program feasibility.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
47 / 68
New Techniques
Object Reuse Methodology
Results Table: Sources of individuals. With Object Reuse Object Reuse Reverse Object Reuse Op. / Mutation Reverse Object Reuse Op. / Crossover Reproduction
Op. Op. Op. Op.
Without Object Reuse
(25%) (25%) (25%) (25%)
Mutation Op. (34%) Crossover Op. (33%) Reproduction Op. (33%)
Table: Percentage of runs attaining full coverage (%f ) and average number of individuals evaluated per run (#i). MUT
put(Object,Object) putAll(Map) remove(Object) containsValue(Object) get(Object) lastIndexOf(Object,int) indexOf(Object,int) removeElementAt(int) addAll(int,Collection) remove(int)
José Carlos Bregieiro Ribeiro (IPL)
CCN
With OR %f #i
W/out OR %f #i
Random %f #i
10 10 3 3 2
10% 85% 25% 100% 25%
4563 1389 4119 501 4000
0% 95% 0% 100% 0%
5000 1154 5000 548 5000
0% 75% 0% 100% 0%
5000 2385 5000 628 5000
10 8 6 5 4
60% 40% 85% 100% 90%
3203 4243 1829 871 1904
0% 0% 75% 95% 80%
5000 5000 2258 1130 2545
0% 0% 70% 80% 80%
5000 5000 2948 1668 2815
Ph.D. Thesis Presentation
12th November, 2010
48 / 68
New Techniques
Object Reuse Methodology
Results
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
49 / 68
Conclusions
Outline 1 Introduction
Background Related Work 2 Methodology Overview
The eCrash Framework 3 New Techniques
Fitness Evaluation Strategy Input Domain Reduction Technique Adaptive Evolutionary Testing Approach Object Reuse Methodology 4 Conclusions
Future Work 5 Tool Demonstration José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
50 / 68
Conclusions
Conclusions
•
This Thesis presented a GP-based approach to the generation of structural Unit Test data for Object-Oriented software.
•
The initial objectives of this work were those of: • positively contributing to improving the level of automation and
performance of the Software Testing process; and • investigating the pertinence of applying Evolutionary Algorithms to
Test Data generation problems. •
It is possible to affirm that these goals were achieved: • the techniques proposed enable enhancing both the efficiency and the
effectiveness of Object-Oriented Evolutionary Testing approaches; and • the development of the eCrash automated tool demonstrates the
applicability of GP to OOET problems. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
51 / 68
Conclusions
Conclusions
•
Test Data Generation is a complex, especially if the goal is to implement an automated and adaptable solution.
•
Methodologies that promote the coverage of problematic structures and difficult control-flow paths are required.
•
We proposed tackling this particular challenge by constantly adapting the direction of the search. • This strategy causes the fitness of feasible Test Programs to fluctuate
throughout the search process, • and allows unfeasible Test Programs to be considered at certain
points of the evolutionary search. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
52 / 68
Conclusions
Conclusions •
The Input Domain Reduction methodology proposed allows eliminating irrelevant variables. • It provides a means to automatically identify and remove Function Set
entries that do not contribute to the definition of test scenarios. • and also ensures that Test Programs are not rendered unfeasible by
the inclusion of uninteresting instructions. •
The inclusion of relevant instructions into the Test Programs provides the rationale for the adaptive methodology presented. • This strategy dynamically updates the selection probabilities of the
constraints defined in the Function Set. • It obtains feedback from the solutions previously produced and
evaluated in order to promote diversity and Test Program feasibility.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
53 / 68
Conclusions
Conclusions
•
The Object Reuse strategy presented enables traversing structures that would not be reachable otherwise. • It effectively enables the reuse of the object references returned by
the functions corresponding to the reused sub-trees, • and is thus especially important if White-Box adequacy criteria are
employed. •
The eCrash tool embodies the Evolutionary Testing approach. • It was implemented during the course of the research, • and allowed studying and experimenting with novel methodologies for
improving the Evolutionary Testing process.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
54 / 68
Conclusions
Future Work
Future Work
•
Near-future plans involve publishing an IDE-integrated version of eCrash – which can be used by Software Testers in a production environment and Evolutionary Testing researchers alike.
•
We will also address some of our approache’s limitations: • the testing and structural coverage of non-public methods; • the Oracle generation problem.
•
We are also planning to experiment with: • parallel systems; • Multi Objective Optimization; • search space sampling.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
55 / 68
Tool Demonstration
Outline 1
Introduction Background Related Work
2
Methodology Overview The eCrash Framework
3
New Techniques Fitness Evaluation Strategy Input Domain Reduction Technique Adaptive Evolutionary Testing Approach Object Reuse Methodology
4
Conclusions Future Work
5
Tool Demonstration José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
56 / 68
Tool Demonstration
The eCrash Automated Testing Tool
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
57 / 68
Tool Demonstration
eCrash – the Gunners Version
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
59 / 68
Appendix
Outline
6
Appendix Scientific Production References
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
61 / 68
Appendix
Scientific Production
Scientific Production I Journal Articles: José Carlos Bregieiro Ribeiro, Mário Alberto Zenha-Rela, and Francisco Fernández de Vega. Test case evaluation and input domain reduction strategies for the evolutionary testing of object-oriented software. Journal of Information and Software Technology, 51(11):1534–1548. Elsevier, 2009. 2009 Impact Factor: 1.821
Conference Proceedings: José Carlos Bregieiro Ribeiro, Mário Alberto Zenha-Rela, and Francisco Fernández de Vega. Adaptive evolutionary testing: an adaptive approach to search-based test case generation for object-oriented software. In NICSO 2010 - International Workshop on Nature Inspired Cooperative Strategies for Optimization, Studies in Computational Intelligence. Springer, 5 2010. José Carlos Bregieiro Ribeiro, Mário Alberto Zenha-Rela, and Francisco Fernández de Vega. Enabling object reuse on genetic programming-based approaches to object-oriented evolutionary testing. In EuroGP 2010 - 13th European Conference on Genetic Programming, Lecture Notes in Computer Science. Springer, 4 2010. José Carlos Bregieiro Ribeiro, Mário Zenha-Rela, and Francisco Fernández de Vega. A strategy for evaluating feasible and unfeasible test cases for the evolutionary testing of object-oriented software. In AST ’08: Proceedings of the 3rd International Workshop on Automation of Software Test, pages 85–92, New York, NY, USA, 2008. ACM. José Carlos Bregieiro Ribeiro, Mário Alberto Zenha-Rela, and Francisco Fernández de Vega. Strongly-typed genetic programming and purity analysis: input domain reduction for evolutionary testing problems. In GECCO ’08: Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation, pages 1783–1784, New York, NY, USA, 7 2008. ACM.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
62 / 68
Appendix
Scientific Production
Scientific Production II Ribeiro, J. Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming. In Proc. of the GECCO ’08, pp. 1819-1822, Graduate Student Workshop, Atlanta, Georgia, USA, July 2008. Ribeiro, J. and Rela, M. Z. and Vega, F.F. Strongly-Typed Genetic Programming and Purity Analysis: Input Domain Reduction for Evolutionary Testing Problems. In Proc. of the GECCO ’08, pp. 1783-1784, 10th Annual Conference on Genetic and Evolutionary Computation, Atlanta, Georgia, USA, July 2008. Ribeiro, J. and Rela, M. Z. and Vega, F.F. A strategy for evaluating feasible and unfeasible test cases for the evolutionary testing of object-oriented software. In Proc. of the 3rd International Workshop on Automation of Software Test (AST’08), pp. 85-92, 30th International Conference on Software Engineering (ICSE’08), Leipzig, Germany, May 2008. Ribeiro, J. and Rela, M. Z. and Vega, F.F. An Evolutionary Approach For Performing Structural Unit-Testing On Third-Party Object-Oriented Java Software. In Proc. of the NICSO 2007, pp. 379-388, Vol. 129/2008, Studies in Computational Intelligence, International Workshop on Nature Inspired Cooperative Strategies for Optimization, Acireale, Italy, November 2007. Ribeiro, J. and Rela, M. Z. and Vega, F.F. eCrash: a Framework for Performing Evolutionary Testing on Third-Party Java Components. In Proc. of the I Jornadas sobre Algoritmos Evolutivos y Metaheuristicas (JAEM), pp. 137-144, II Congreso Español de Informática (CEDI), Zaragoza, Spain, September 2007. ISBN: 978-84-9732-593-6. Ribeiro, J. and Vega, F.F. and Rela, M. Z. Using Dynamic Analysis of Java Bytecode for Evolutionary Object-Oriented Unit Testing. In Proc. of the 8th Workshop on Testing and Fault Tolerance (WTF), pp. 143-156, 25th Brazilian Symposium on Computer Networks and Distributed Systems (SBRC), Belém, Brazil, May 2007. ISBN: 85-766-0119-1.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
63 / 68
Appendix
Scientific Production
Scientific Production III Ribeiro, J. and Luis, B.M. and Rela, M. Z. Error propagation monitoring on windows mobile-based devices. In Proc. of the LADC 2007, pp. 111-122, Vol. 4746/2007, Lecture Notes in Computer Science, Latin-American Symposium on Dependable Computing, Morelia, Mexico, September 2007. Ribeiro, J. and Rela, M. Z. mCrash: a Framework for the Evaluation of Mobile Devices Trustworthiness Properties. In Proc. of the CSMU 2006, pp. 163-166, Conference on Mobile and Ubiquitous System, Guimarães, Portugal, June 2006. ISBN: 972-8692-29-3.
Theses: Master in Informatics and Systems mCrash: a Framework for the Evaluation of Mobile Devices’ Trustworthiness Properties. Organization: University of Coimbra, Portugal; Supervisor: Prof. Mário Zenha-Rela; Period: 2005-2008; Presentation Date: 17th of December, 2008 Diploma de Estudios Avanzados Evolutionary Unit-Testing Of Third-Party Object-Oriented Java Software. Organization: University of Extremadura, Spain; Supervisor: Prof. Francisco Fernández de Vega; Period: 2006-2007; Presentation Date: 29th of November, 2007
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
64 / 68
Appendix
References
References I [Arc08] Andrea Arcuri. On the automation of fixing software bugs. In ICSE Companion ’08: Companion of the 30th international conference on Software engineering, pages 1003–1006, New York, NY, USA, 2008. ACM. [AY07a] Andrea Arcuri and Xin Yao. Coevolving programs and unit tests from their specification. In ASE ’07, pages 397–400, New York, NY, USA, 2007. ACM. [AY07b] Andrea Arcuri and Xin Yao. A memetic algorithm for test data generation of object-oriented software. In CEC’07, pages 2048–2055. IEEE, 2007. [CKP05] Yoonsik Cheon, Myoung Kim, and Ashaveena Perumandla. A complete automation of unit testing for java programs. In Hamid R. Arabnia and Hassan Reza, editors, SERP’05, Las Vegas, Nevada, USA, June 27-29, 2005, Volume 1, pages 290–295. CSREA Press, 2005. [DJAR07] C. S. Siva Dharsana, D. Nithila Jennifer, A. Askarunisha, and N. Ramaraj. Java based test case generation and optimization using evolutionary testing. In ICCIMA ’07, pages 44–49, Washington, DC, USA, 2007. IEEE Computer Society. [GR08] Nirmal Kumar Gupta and Mukesh Kumar Rohil. Using genetic algorithm for unit testing of object oriented software. In ICETET ’08, pages 308–313, Washington, DC, USA, 2008. IEEE Computer Society. [IX07] Kobi Inkumsah and Tao Xie. Evacon: A framework for integrating evolutionary and concolic testing for object-oriented programs. In ASE 2007, pages 425–428, November 2007.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
65 / 68
Appendix
References
References II [IX08] Kobi Inkumsah and Tao Xie. Improving structural testing of object-oriented programs via integrating evolutionary testing and symbolic execution. In ASE 2008, September 2008. [KDR07] Alex Kinneer, Matthew B. Dwyer, and Gregg Rothermel. Sofya: Supporting rapid development of dynamic program analyses for java. In ICSE COMPANION ’07: Companion to the proceedings of the 29th International Conference on Software Engineering, pages 51–52, Washington, DC, USA, 2007. IEEE Computer Society. [LR08] Konstantinos Liaskos and Marc Roper. Hybridizing evolutionary testing with artificial immune systems and local search. In ICSTW ’08, pages 211–220, Washington, DC, USA, 2008. IEEE Computer Society. [LRW07] Konstantinos Liaskos, Marc Roper, and Murray Wood. Investigating data-flow coverage of classes using evolutionary algorithms. In GECCO ’07, pages 1140–1140, New York, NY, USA, 2007. ACM. [LWL05] Xiyang Liu, Bin Wang, and Hehui Liu. Evolutionary search in the context of object-oriented programs. In MIC’05, 2005. [MS76] W. Miller and D. L. Spooner. Automatic generation of floating-point test data. IEEE Trans. Softw. Eng., 2(3):223–226, 1976. [SAY07] Ramon Sagarna, Andrea Arcuri, and Xin Yao. Estimation of distribution algorithms for testing object oriented software. CEC’07, Singapore, 25-28 September 2007. IEEE Computational Intelligence Society, IEEE Press. [See06] Arjan Seesing. Evotest: Test case generation using genetic programming and software analysis. Master’s thesis, Delft University of Technology, 2006. José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
66 / 68
Appendix
References
References III [SG06] Arjan Seesing and Hans-Gerhard Gross. A genetic programming approach to automated test generation for object-oriented software. International Transactions on System Science and Applications, 1(2):127–134, 2006. [Ton04] Paolo Tonella. Evolutionary testing of classes. In ISSTA ’04, pages 119–128, New York, NY, USA, 2004. ACM Press. [Wap07] Stefan Wappler. Automatic Generation Of Object-Oriented Unit Tests Using Genetic Programming. PhD thesis, Technischen Universitat Berlin, 12 2007. [WS07] Stefan Wappler and Ina Schieferdecker. Improving evolutionary class testing in the presence of non-public methods. In ASE ’07, pages 381–384, New York, NY, USA, 2007. ACM. [WW06a] S. Wappler and J. Wegener. Evolutionary unit testing of object-oriented software using a hybrid evolutionary algorithm. In CEC’06, pages 851–858. IEEE, 2006. [WW06b] Stefan Wappler and Joachim Wegener. Evolutionary unit testing of object-oriented software using strongly-typed genetic programming. In GECCO ’06, pages 1925–1932, New York, NY, USA, 2006. ACM Press. [XES+92] S. Xanthakis, C. Ellis, C. Skourlas, A. Le Gall, and S. Katsikas andd K. Karapoulios. Application of genetic algorithms to software testing. In Proceedings of the 5th International Conference on Software Engineering, pages 625–636, 1992. [XTdHS08] Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. Method-sequence exploration for automated unit testing of object-oriented programs. In SSEAT’08, July 2008.
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
67 / 68
The End
Contributions for Improving Genetic Programming-Based Approaches to the Evolutionary Testing of Object-Oriented Software by José Carlos Bregieiro Ribeiro Supervisors: F. Fernández de Vega Mário Zenha-Rela Mérida, Spain 12th November, 2010
José Carlos Bregieiro Ribeiro (IPL)
Ph.D. Thesis Presentation
12th November, 2010
68 / 68
