DECISION-THEORETIC PLANNING
A DISSERTATION SUBMITTED TO THE DEPARTMENT OF ENGINEERING-ECONOMIC SYSTEMS AND OPERATIONS RESEARCH AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
Mark Alan Peot June 1998
i
Copyright © by Mark Alan Peot 1998 All Rights Reserved
ii
I certify that I have read this dissertation and that in my opinion it is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy.
_______________________________ Ross D. Shachter (Principal Adviser)
I certify that I have read this dissertation and that in my opinion it is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy.
_______________________________ David E. Smith
I certify that I have read this dissertation and that in my opinion it is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy.
_______________________________ Edison T. S. Tse
Approved for the University Committee on Graduate Studies:
________________________________
iii
Abstract In recent years, researchers in AI planning have been trying to extend the classic planning paradigm to handle uncertainty and partial goal satisfaction. This dissertation focuses on the development of two partial-order planners: DTPOP, a contingent decision-theoretic planner, and UDTPOP, its non-contingent sibling. Both planners assemble actions with conditional, uncertain effects into plans of high overall utility.
A number of innovations are claimed for UDTPOP. UDTPOP uses a new criterion for identifying actions that are relevant to utility. Steps are constrained to be effective, that is, they are guaranteed to change the world in ways that can result in high utility outcomes. In nearly every other classical partial-order planner, an action can support an open precondition only if it has an outcome that can directly support that precondition. In UDTPOP support can be indirect as long as the added step supports desirable utility outcomes and doesn’t cause any other step to lose effectiveness. This new action selection criterion allows us to abandon the multiple support paradigm of the probabilistic planner Buridan. This, in turn, allows us to design a nearly systematic planner. The same innovation also allows us to develop a strict (and tight) upper bound on the utility of the plan that can be used as an admissible cost function for best first search. UDTPOP is significantly more effective than previous methods. An empirical study compares a variant of UDTPOP against Buridan. In nearly every domain, the performance of UDTPOP is superior, sometimes by 2-3 orders of magnitude.
The second planner, DTPOP, is the first sound and complete contingent partial-order planner. Efficient contingency plans satisfy a “no augury” condition: information used to decide between plan branches must be relevant (in the sense of D-separation) to the utility of the branches under consideration. DTPOP uses this criterion directly to develop tests that can reveal the distribution over unknown, but relevant, events. A test is a subplan that has observable outcomes that are dependent on these hidden events. During planning, DTPOP reverses its direction of search, using regression to establish preconditions and progression to identify observations.
iv
Acknowledgements This work was funded in part through DARPA contract F30602-91-C-0031, the Rockwell International Science Center, and by the National Science Foundation through a Graduate Fellowship.
The ideas in this dissertation benefitted considerably from discussions with my advisor Ross Shachter, Jack Breese, Denise Draper, Ken Fertig, Nir Friedman, Steve Hanks, David (DE2) Smith, Phil Stubblefield, Edison Tse, Dan Weld & Michael Wellman. The ideas in this dissertation arose through joint work performed with DE2 Smith and Jack Breese. Jim Martin and the Rockwell Palo Alto Laboratory provided the perfect working environment for my research.
Finally, I want to thank my friends, Brian Gregory, Michael Lippitz, Enrique Romero, David Smith, and Tim Stanley, for putting up with a lot of dissertation-related flakiness over the past years. Thanks also to my parents, Hans and Kathleen Peot, for their considerable moral support over the years.
My true love Lydia Chang managed my dissertation progress from Michigan.
v
vi
Table of Contents
Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii List of Illustrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv Notation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi 1.0
2.0
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1
The Problem
1
1.2
The Solution
4
1.2.1
Non-contingent Solutions: UDTPOP
4
1.2.2
DTPOP
4
1.3
Roadmap of Regression Planner Development
5
1.4
Contributions
7
1.5
Outline of this dissertation
10
Action and Plan Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.1
Variables and Functions
14
2.2
Steps
15
2.2.1
Conditional Effects
15
2.2.2
Conditional Cost Models
16
2.2.3
Belief Network Representation for Steps
17
2.2.4
Example: Stress World
18
2.2.5
Frame Assumptions
21
2.3
Goals and Utility
22
2.4
Assembling Steps into Plans
23
2.5
Example
26
2.6
Related Work
27
2.6.1
Attributes of Probabilistic Steps
27
vii
3.0
2.6.2
Attribute Cardinality
27
2.6.3
World State Representation
30
2.6.4
Distribution Symmetry
31
2.6.5
Distribution Completeness
32
UDTPOP: Noncontingent Planning. . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.1
Overview
35
3.2
The Basic Ideas
36
3.2.1
Multiple Support
36
3.2.2
Single Support
40
3.2.3
Effectiveness
47
3.2.3.1
Possibility
48
3.2.3.2
Pertinence
48
3.2.3.3
Effectiveness
49
3.3
UDTPOP
53
3.3.1
Plan Flaws: Open Conditions
55
3.3.2
Plan Flaws: Threats
56
3.3.3
Adding Support: Add-Step and Add-Link
56
3.3.4
Resolving Threats: Promote and Demote
58
3.3.5
Resolving Threats: Persist-Support
60
3.4
Example
62
3.5
Approximating Effectiveness
67
3.5.1
Possibility
67
3.5.2
Pertinence
68
3.5.3
Effectiveness
69
Evaluating Plans
70
3.6 3.6.1
Search
70
3.6.2
Model Construction in Complete Plans
72
3.6.3
Model Construction in Partial Plans
78
3.6.3.1
Modeling Open Conditions
79
3.6.3.2
Modeling Open Conditions Using LPE
81
3.6.3.3
Modeling Threats
89
3.6.3.4
Model Construction and Evaluation for Partial Plans
91
3.6.3.5
Persist-Support
95
3.6.4
3.7
The Implementation of the Evaluator used in UDTPOP-B
Formal Properties
3.7.1 3.7.1.1
Soundness Markov Model
97
98 99 100
viii
3.7.1.2 3.7.2
Soundness Completeness
102
3.8
Empirical Results
103
3.9
Discussion
114
3.9.1
Mutual Exclusion
114
3.9.2
Comparison with Buridan
115
3.9.2.1
Link Establishment Order
115
3.9.2.2
Backtracking on Open Conditions
116
3.9.2.3
Plan Evaluation and Mutual Exclusion
117
3.9.2.4
Persist-Support vs. Confrontation
118
3.9.2.5
Soundness
118
3.9.2.6
The Buridan Heuristic
119
3.10 Extensions
122
3.10.1
Extending Relevance
122
3.10.2
Extending Effectiveness
123
3.10.3
Tightening the Upper Bound on Evaluation
124
3.10.4
Systematicity
125
3.10.5
Multiple Support and Ordering Constraints
126
3.10.5.1
Multiple Support = Fewer Ordering Constraints (sometimes...)
126
3.10.5.2
Commutivity
127
3.11 Contributions
4.0
128
Relevance and Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 4.1
Notation and Definitions
132
4.1.1
Belief Network Notation
132
4.1.2
Influence Diagrams
133
4.2
Observation Relevance
134
4.3
Identifying Relevant Nodes in Belief Networks
137
4.3.1
The Ne, Np, Nr and Ni sets
137
4.3.2
The Bayes Ball Algorithm
139
4.3.3
Ne, Np, Ni, and Nr Examples
142
4.3.3.1
Collect-Requisite
142
4.3.3.2
Collect Relevant
144
4.4
5.0
102
Dynamic Influence Diagrams
146
DTPOP: Contingent Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 5.1
Overview
151
5.2
Basic Ideas
152
5.2.1
Constructing Contingent Plans
152
ix
5.2.1.1
Warplan-C
153
5.2.1.2
CNLP
155
5.2.1.3
C-Buridan
158
5.2.1.4
Cassandra
162
5.2.1.5
DTPOP
163
5.2.2
5.3
Identifying Observations
DTPOP
165
168
5.3.1
DTPOP Plan Elements
171
5.3.2
Mutual Exclusion Constraints and Execution Policies
172
5.3.3
Plan Flaws
174
5.3.3.1
Plan Flaws: Threats
174
5.3.3.2
Open Uncertainties
175
5.3.4
UDTPOP Modifications
177
5.3.5
Threat Resolution: Branch
177
5.3.6
Discovering Alternative Plans: Add-Branch
178
5.3.7
Identifying Observations: Add-Link-Forward and Add-Step-Forward
178
5.3.8
Remove-Open-Uncertainty
180
5.4
Plan Optimization
185
5.4.1
Contingent Plans and Asymmetric Influence Diagrams
186
5.4.2
Plan Optimization
187
5.4.2.1
Plan Model Construction
188
5.4.2.2
Pass 1
191
5.4.2.3
Pass 2
195
5.4.2.4
Plan Optimization
195
5.4.2.5
Simplifying the Decision Problem
196
5.4.3
5.5
Approximating Plan Optimization
198
Recognizing Open Uncertainties
204
5.5.1
Background
204
5.5.2
The Information Relevance Network
205
5.6
Heuristics
214
5.6.1
Inter-branch threats
214
5.6.2
Closing and Constraining Open-Uncertainties
215
5.7
Example
216
5.7.1
The First Plan Branch
218
5.7.2
Making the Initial Plan More Robust
220
5.7.3
Inter-Branch Threats
222
5.7.4
Searching for Observations
224
5.7.5
Constructing the Decision Tree.
225
x
5.7.6
5.8
Solving the Decision Tree
Formal Properties
229
232
5.8.1
Soundness
232
5.8.2
Completeness
233
5.9
Discussion
5.9.1
Related Work
234 234
5.9.1.1
Causal Link Planners
234
5.9.1.2
Markov Decision Processes
237
5.9.1.3
MAXPLAN
241
5.9.1.4
Knowledge-Based Model Construction
242
5.9.2
Knowledge Preconditions
242
5.9.3
Observation Actions
246
5.9.4
Classes of Plans and Independence
248
5.10 Extensions and Conclusions
250
5.10.1
Experimental Validation
250
5.10.2
Evaluation of Partial Plans
251
5.10.3
Multiplicative Growth in Search Space with Plan Branches.
252
5.11 Contributions
253
6.0
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
A.
UDTPOP Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 A.1
Effective Support
273
A.2
Pertinence
275
A.3
Soundness
276
A.4
Completeness
281
A.4.1
Identifying Causal Structure
281
A.4.2
The Clairvoyant Decision Policy and PertinentC
283
A.4.3
Completeness
288
A.5
B.
Admissible Upper Bound
292
DTPOP Proofs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 B.1
Notation
299
B.2
Eliminating Decisions
299
B.3
Observation Relevance
300
B.4
Soundness
302
B.5
Completeness
305
B.5.1
Identifying the Contingent Plan
305
B.5.2
No Fusion
308
xi
B.5.3
Completeness
309
xii
List of Tables
TABLE 2. The drink-coffee step.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
TABLE 3. The effect of the write step on total number of pages written. TABLE 4. The effect of the Write step on Harvey’s state of alertness. TABLE 12.
UDTPOP-B vs. Buridan-R and Buridan-F
TABLE 13.
Empirical Comparison in a Navigation Domain
. . . . . . . . . . . . . . . . . . . . . . . . . 20 . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
xiii
List of Illustrations
FIGURE 1.
Roadmap of Regression Planner Development
..................... 6
FIGURE 2.
Relative performance of Buridan and UDTPOP-B
FIGURE 3.
Step Model
FIGURE 4.
Steps in Stress World
FIGURE 5.
Causal Links.
FIGURE 6.
A short plan
FIGURE 7.
A chain of causal links protecting attribute variable .
FIGURE 8.
Asymmetric conditional effects
FIGURE 9.
A Buridan step
FIGURE 10.
Increasing the probability of a precondition with a causal link
FIGURE 11.
Increasing the probability of a precondition with multiple links from the same step 39
FIGURE 12.
Increasing the probability of a precondition using a causal link from a different step 40
FIGURE 13.
Single Support
FIGURE 14.
Increasing the probability of a precondition using a causal link from a different step 42
................... 9
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 . . . . . . . . . . . . . . . . 29
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 . . . . . . . . . 39
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
xiv
FIGURE 15.
Choices in Single and Multiple Support
. . . . . . . . . . . . . . . . . . . . . . . . . . 44
FIGURE 16.
Proliferation of Structure in Multiple Support Planners
FIGURE 17.
A very simple navigation domain.
FIGURE 18.
A “state transition” diagram for the “Go North” step.
FIGURE 19.
Constraints.
FIGURE 20.
Effectiveness
FIGURE 21.
Threats
FIGURE 22.
Promotion
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
FIGURE 23.
Demotion
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
FIGURE 24.
Before Persist-Support
FIGURE 25.
After Persist-Support
FIGURE 26.
Initial Plan:
FIGURE 27.
The example after adding SW1
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
FIGURE 28.
The example after adding SW2
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
FIGURE 29.
The example after adding a link from SIC to SW2
. . . . . . . . . . . . . . . . . . 65
FIGURE 30.
The example after adding a link from SIC to SW1
. . . . . . . . . . . . . . . . . . 65
FIGURE 31.
The example after persist-support
FIGURE 32.
The final complete plan
FIGURE 33.
Best First Search.
FIGURE 34.
Model_CE
. . . . . . . . . . . . . . . 45
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 . . . . . . . . . . . . . . . . . 52
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
xv
FIGURE 35.
A Model for a Complete Plan
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
FIGURE 36.
Model_CE(RGoal, SGoal)
FIGURE 37.
Model_CE(Pages(SW1+), SW1).
FIGURE 38.
Trace of Model_CE
FIGURE 39.
Trace of Model_CE continued
FIGURE 40.
Trace for Model_CE completed
FIGURE 41.
A fallacious argument for using decisions for representing open conditions 80
FIGURE 42.
A simple partial plan that breaks the ‘straw’ model construction algorithm
FIGURE 43.
Using SA to support both preconditions maximizes utility
FIGURE 44.
LPE Example I
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
FIGURE 45.
LPE Example II
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
FIGURE 46.
Active Set 1
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
FIGURE 47.
Active Set 2
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
FIGURE 48.
Active Set 3
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
FIGURE 49.
Modeling every completion of a partial plan
FIGURE 50.
EvaluateUB
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
FIGURE 51.
Model_CE2
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
FIGURE 52.
UB
FIGURE 53.
Persist Support can cause dual support
FIGURE 54.
The contradiction in the proof for Theorem 8
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
81
. . . . . . . . . . . . 81
. . . . . . . . . . . . . . . . . . . . . . . 89
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 . . . . . . . . . . . . . . . . . . . . . . . . . . 95 . . . . . . . . . . . . . . . . . . . . . . 97
xvi
FIGURE 55.
The cluster tree constructed implicitly by the UDTPOP evaluator
. . . . . . 98
FIGURE 56.
Markov Model.
FIGURE 57.
The Markov model for a 2 step plan.
FIGURE 58.
Relative Performance of UDTPOP-B and Buridan-R
. . . . . . . . . . . . . . . 108
FIGURE 59.
Relative Performance of UDTPOP-B and Buridan-F
. . . . . . . . . . . . . . . 109
FIGURE 60.
The branching factor of Buridan and UDTPOP-B as a function of search space depth in the Mocha Blocks World 0.899 domain 111
FIGURE 61.
The branching factor of Buridan and UDTPOP-B as a function of search space depth in Diamond World 111
FIGURE 62.
The relative search space sizes for Buridan-R and UDTPOP-B in Mocha Blocks World 0.899 112
FIGURE 63.
The relative search space sizes for Buridan-R and UDTPOP-B in Diamond World 114
FIGURE 64.
Simple Link Domain
FIGURE 65.
The network of roads used for the navigation domain.
FIGURE 66.
The Move Operator
FIGURE 67.
Multiple support can result in fewer ordering constraints.
FIGURE 68.
The Dry-With-Wet-Towel Action from Wet Towel World
FIGURE 69.
Influence Diagram
FIGURE 70.
A simple decision problem with one decision and no observations
FIGURE 71.
A simple decision problem with one observation
FIGURE 72.
Irrelevant observations
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 . . . . . . . . . . . . . . 121
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 . . . . . . . . . . . . 127 . . . . . . . . . . . . . 127
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 . . . . 135
. . . . . . . . . . . . . . . . . . 136
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
xvii
FIGURE 73.
Bayes_Ball
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
FIGURE 74.
Collect-Requisite
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
FIGURE 75.
Collect-Relevant
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
FIGURE 76.
Collect Requisite Cases.
FIGURE 77.
Relevance Examples
FIGURE 78.
Planning in Warplan-C
FIGURE 79.
Planning in CNLP
FIGURE 80.
Outcomes for a gluing operation.
FIGURE 81.
Observation Plan Construction in DTPOP
FIGURE 82.
The top level of the DTPOP planning algorithm
FIGURE 83.
Threats
FIGURE 84.
Voltage Measurement Example
FIGURE 85.
Branch
FIGURE 86.
Inefficient Plans
FIGURE 87.
Remove-Open-Uncertainty Cases I and 2
. . . . . . . . . . . . . . . . . . . . . . . 183
FIGURE 88.
Remove-Open-Uncertainty Cases 3 and 4
. . . . . . . . . . . . . . . . . . . . . . . 184
FIGURE 89.
Plan Optimization
FIGURE 90.
PM
FIGURE 91.
PM_1
FIGURE 92.
Model_Step
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 . . . . . . . . . . . . . . . . . . . . . . . 167 . . . . . . . . . . . . . . . . . . . 170
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
xviii
FIGURE 93.
Model_CE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
FIGURE 94.
Observability and Model_CE
FIGURE 95.
Model_CL
FIGURE 96.
An algorithm for systematically generating partial orders with equivalent value topological sorts 198
FIGURE 97.
A Decision Tree
FIGURE 98.
Build-Tree
FIGURE 99.
Prune-Mutex
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
FIGURE 100.
Build-Action
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
FIGURE 101.
Merge
FIGURE 102.
Active Subgraph Example, Part I
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
FIGURE 103.
Active Subgraph Example, Part II
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
FIGURE 104.
The VOM calibration example
FIGURE 105.
There are possibly many active paths for a single relevance relation
FIGURE 106.
Relevance For Observable Nodes.
FIGURE 107.
Collect-Relevant
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
FIGURE 108.
Collect-Requisite
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
FIGURE 109.
Bayes_Ball_IRN
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
FIGURE 110.
A non-contingent plan for partying outdoors.
FIGURE 111.
Expected Value of Party-Outdoors
FIGURE 112.
The plan after starting another plan branch.
FIGURE 113.
The plan after completing the second plan branch.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 . . . 209
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
. . . . . . . . . . . . . . . . . . . . . 220
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 . . . . . . . . . . . . . . . . . . . . . . 221 . . . . . . . . . . . . . . . . 221
xix
FIGURE 114.
Expected Value for the party alternatives
. . . . . . . . . . . . . . . . . . . . . . . . 222
FIGURE 115.
Inter-branch threats.
FIGURE 116.
Mutex Sets After Branching
FIGURE 117.
The Information Relevance Network
FIGURE 118.
The Information Relevance Network after Add-Step-Forward
FIGURE 119.
One possible topological sort for the contingent plan
FIGURE 120.
The Full Decision Tree Constructed by Build-Tree
FIGURE 121.
Decision Tree after Marginalizing Out the Unobserved Variables
FIGURE 122.
Decision Maximization
FIGURE 123.
Marginalization of Observable Variables
FIGURE 124.
Value of the contingent plan
FIGURE 125.
Simple Forecast Example
FIGURE 126.
Multiple Forecasts
FIGURE 127.
The transformation from Mm to Mm’
FIGURE 128.
Discover Links.
FIGURE 129.
Action schemata for SA and SB.
FIGURE 130.
Constructing Q
FIGURE 131.
Belief networks for Case I
FIGURE 132.
Gadget for Representing a Disjunct in 3SAT
FIGURE 133.
Representation for 3SAT.
FIGURE 134.
Discover_Contingent_Links
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 . . . . . . . . 225
. . . . . . . . . . . . . . . 226
. . . . . . . . . . . . . . . . . 229 . . . . . . 230
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 . . . . . . . . . . . . . . . . . . . . . . . . . 231
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 . . . . . . . . . . . . . . . . . . . . . 301
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
xx
Notation A, B, Variable : A, B, Variables :
Attribute variables (Generalized propositions). Sets of attribute variables.
a, b, value ( a, b, values ) : S 1, S A, S Go : S i - ( S i+ ) :
V
ordering constraint, “ S 1 completes before S 2 commences.”
: A causal link from S E to S C protecting attribute variable V .
ST ⊗ L : ST
: in a conditional effect distribution CE is COs(CE) . are conditional outcomes of this step because the each , where S, . Each trigger, c i , of each conditional outcome is a set of propositions. The triggers for the set of conditional outcomes are mutually exclusive ( ∀c i ≠ c j , c i ∧ c j = ⊥ ) and collectively exhaustive ( c 1 ∨ … ∨ c n = T ). Each effect e i is a set of propositions denoting the set of changes that are made to the state of the world if that outcome is the result of step execution. Any e k can be empty, indicating that conditional outcome has no effect on the world state. The set of conditional effects defining a step can be represented as a probability tree. The leaves of this tree indicate the individual outcomes ( e i ). The path from the root of the tree to each leaf encodes the trigger, c i , of each conditional outcome. Figure 9 illustrates a hypothetical drink-coffee step. If the coffee is caffeinated, drink-coffee changes the drinker’s level of alertness to conscious with an 80% probability (Leaf 1 in Figure 9) and has no effect at all with a 20% probability (Leaf 2). If the coffee is decaffeinated, drinkcoffee results in consciousness with a probability of only 5% and has no effect 95% of the time. 1 {Alert=conscious} p=0.8 is possible when there is some completion of the partial plan in which the probability of its trigger, c , is greater than zero. is effec- is passive with respect to variable A if ∃i, j such that A = E i = C j and ei = c j . , where R is the reward utility function, A IC , Flaws ) if there exists a step S E in the domain description that has Open in its effect variables, return Add-Support( oc , S E , , Flaws ) else return ∅ . , Flaws ) if there exists a step S E in S such that S E is possibly before S C and Open is an effect variable of S E , return Add-Support( oc , S E , P , Flaws ) else return ∅ , Flaws ) if (there is a conditional outcome ce = of step S E such that 1. ∃j such that Open = E j , 2. e j is pertinent for precondition Open of step S C 3. c is possible, and 4. ce is effective for Open.) then { Open let K new := effective(S E → S C ) Open let P' := let Flaws' := Prune(Flaws) ∪ newOCs P(S E, Open, P) ∪ newThreats(P) return Complete-Plan( P′ , Flaws' ) } else return ∅ , Flaws ) If ( S C < S T is possible){ O′ := O + ( S C < S T ) return Complete-Plan( , Prune(Flaws) ) } else return ∅ . , Flaws ) If ( S T < S E is possible){ O′ := O + ( S T < S E ) return Complete-Plan(, Prune(Flaws) ) else return ∅ . , Flaws ) if (there exists a conditional outcome ce = in S T such that ∃j such that Ej = V, ce is a passive conditional outcome for V, c is possible, ej is a pertinent value for V.) V V V then Let L′ := L – ( S E →S C ) + ( S E →S T ) + ( S T →S C ) O' := O + ( S E < S T ) + ( S T < S C ) V K new := persist(S T →S C) 13 P' = Flaws′ = Prune(Flaws) ∪ NewOCs(S T, V, P′) return Complete-Plan( P′ ) and . of S such that ∀c j ∈ c, possible P(c i) . 3.5.2 and e is pertinentP. is an effective conditional outcome of S T , and c is possible , then V = v cannot be a pertinent p precondition of S C . This is a restatement of the clairvoyant decision policy lemma (Lemma 3) used by the completeness proof. The basic idea behind the proof (the proof is actually quite a bit more complicated than this): V and a belief network M = = 0.5 . = 0.5 ) { Let M := <∅, ∅ > //the model (a global variable) Util = R ∪ ( – Cost(S i) ) ) if CE ∉ N then { N := N + CE for all V ∈ PreVars(CE) V { if ∃( S E →Step ∈ L ) then { for all Cost E ∈ Cost(S E) , Model_CE( Cost E , S E ) find CE E ∈ Eff(S E) such that V ∈ OutVars(CE E) Model_CE( CE E , S E ) if V(S E+) ∉ N then { E := EffVars(CE E) V(S E+) := the distribution P { V ( S E+ ) = v V = v, E\V } = 1.0 N := N + V(S E+) A := A + ( CE E, V(S E+) ) } A := A + ( V(S E+), CE ) } else { // V is an open condition if ∃ a probability node for precondition var V of S then { let O V := be that node } else { let O V := a probability node w/states Domain(V) . //a vacuous distribution. P { O V } = [ 0, 1 ] ) { Let M′ = be the subgraph of M that is in the N p set of U i .[Shachter 98] For all S i ∈ S { if ( ∃ some S j such that S j is possibly after S i and S j has an open precondition) then { Let N S be the set of nodes N that correspond to conditional effect distributions of S i . For all N k ∈ N S { Let X be a vacuous likelihood function P { . N k } = [ 0, 1 ] N′ := N′ + X A = A + ( N k, X ) } } Use LPE to compute the posterior bounds P M′ { U i } . Let u Max := max u u ∈ Uj return max(u Max, , is Q = { S i } i = 1 . Let M m be the ⎛ , where R is the reward function, A is the set of step schema that can be used to construct a plan and IC is the set of distributions over all of the variables used in the domain. Theorem 11 (Completeness): Let Q be a non-empty sequence of steps that is an . The cost functions for the possible steps in A are all greater than zero. UDTPOP with the appropriate search control strategy can identify a partial-order plan P' that has a topological sort that is identical to Q . Proof/ % s. TABLE 12. UDTPOP-B vs. Buridan-R and Buridan-F is a directed acyclic graph (DAG) comprised of a set of probability (or chance) nodes N and a set of directed arcs A. Pa(J) will denote the set of nodes corresponding to the immediate graph predecessors (or parents) of node J. Similarly Ch(J), De(J) and An(J) will denote the children, descendents, and ancestors of J, respectively. Each node J in the belief network represents an uncertain variable with a conditional probability distribution P { J Pa(J) } . Deterministic distributions are conditional probability distributions that consist only of O's and 1's. and sets J, K, L ⊆ N , the set of uncertain variables J is conditionally independent of L given K in belief network B if P { J K, L } = P { J K } . This relationship will be denoted by J ⊥ L K . If the nodes in K are observed and J ⊥ L K , then it is not possible to learn more about the joint distribution of the nodes in J by observing any subset of the nodes in L. The opposite of conditional independence is conditional (or probabilistic) relevance. If ¬( J ⊥ L K ) , then the variables in L are conditionally relevant to the variables in J given observations for the variables in K. is comprised of a set of uncertain nodes, N (denoted by ovals in the figure); a set of directed arcs, A; a set of decision nodes, D ) { //Initialization D := { i ∈ N i is deterministic } K := K in N e := ∅ N p := ∅ N r := ∅ for all j ∈ J in , Collect-Requisite(j,B) N i := ( N – N r ) ∪ K in N r := N r – K in and a bound N; and a bound on the number of steps N, where is the reward utility function; A is a set of uncertain, partially-observable actions; and IC is the distribution over all of the variables prior to the execution of the plan, R . The new field, M , is the set of mutex constraints added by branch. In the second phase of plan construction, DTPOP uses the partial-order plan constructed in the first phase to support the construction a decision tree or influence diagram to compute the optimal step execution policies. The causal links and ordering constraints together and e is pertinentP.. , Flaws ) If ( Reason(S T) ∩ Reason(S C) = ∅ ) then { Complete-Plan( , Prune(Flaws) ) } else { return ∅ } 5.3.6 ) { Let Flaws = { Open ∈ Pre(S G) } , Flaws ) , Flaws ) { if (there exists an action S C in the set of action schema A such that Open is a precondition variable for a conditional effect of S C ) { return Add-Support-Forward( ou , S C , , Flaws ) } else { return ∅ } } , Flaws ) { if (there exists an action S C in S such that S C is possibly after S E , S E is compatible with the reasons of S C , Open is a precondition variable of a conditional effect of S C , and Open S S C or open there is no causal link → E Open condition ( → S C ) ) { return Add-Support-Forward( oc , S C , P , Flaws ) } else { return ∅ } } , Flaws ) { Open L′ := L + ( S E → S C ) let P' := ou new := newOUs(P′) threats new := newThreats(P′) CE obs := a conditional effect that is observable and has Open in its preconditions. pres obs := the preconditions of CEobs. oc link := newOCs P(S E, Open, P′) 21 ⎧ Open ⎫ Open oc obs := ⎨ ( → S C ) Open ∈ pres obs ∧ ( S → S C ) ∉ L′ ⎬ 22 ⎩ ⎭ Flaws' := ( Prune(Flaws) – oc ) ∪ threats new ∪ oc link ∪ oc obs ∪ ou new return Complete-Plan( P′ , Flaws' ) } . The main difference between the structure of this plan and that of a UDTPOP plan lies in the mutex constraints, M . The mutex constraints play three roles in DTPOP: , T = [ S i ] i = 1 ) { B := PM_1(P) //constructs a belief network capturing the //cause and effect relationships between //uncertain variables M := PM_2(B,T) //turns the PM_1 belief network into a //influence diagram. return M } , where N is a set of probability nodes, A is a set of arcs, O ⊆ N is the subset of N that is observable, and V is a set of subvalue nodes. I will abuse the union operator to combine the components of these belief network tuples: B 1 ∪ B 2 = ∪ = ) { B = <∅, ∅, ∅, ∅ > ) { B step = <∅, ∅, ∅, ∅ > for each CE k in Eff(S j) FIGURE 94. Model_CE. Model_CE adds a subvalue node for each cost or reward function in the step. Recall (start of Section 5.5.2) that the reward utility function (and only the reward utility function) must be additively scaled so that the worst possible outcome of the reward function has a value of zero. ) { A := ∅ for each CE k in Eff(S C) { if ( V ∈ PreVars(CE k) and there is a causal link for each precondition of CE k ) { A := A + ( V(S E+), CE k ) } } for each UCE k in Cost(S C) { if ( V ∈ PreVars(UCE k) ) { A := A + ( V(S E+), UCE k ) } } return <∅, A, ∅, ∅ > } ) { //Initialization F := { i ∈ N i is deterministic } K := { i ∈ N i is observable } N top := ∅ N bot := ∅ N r := ∅ Collect-Requisite(V, Di, B) } ; MutexConstraints M) { Open := ∅ loop over all compatible combinations m of values for the mutex constraints { Let B’ := The IRN derived by pruning all of the nodes in B that are not compatible with m. D := the decision nodes in B’. for all D i ∈ D { Let V := The value nodes that are successors of Di for all V i ∈ V { Open := Open ∪ BB_IRN(V j, D i, B′) } } } return Open } . The cost func- does not provide effective support if one of the three cases is true: 1. is not possible, e.g. P { c } = 0 ; 2. is possible, but is not effective; or 3. is possible and effective, but is not pertinent. We can rewrite the sum E(C Sk) as a sum over the three subsets of X(S u+) × X(S u – 1+) corresponding to these three cases (call them L 1 , L 2 , and L 3 , respectively). We will show that the conditional effect of S u can be replaced by a null effect without increasing utility. That is, we will demonstrate that , such that v is , such is Q = { S i } i = 1 . Let M m be the Markov model of Q . The plan model of P , M p , is equivalent to denote the nodes and arcs in M m . Let M m' be the graph . The arcs in M m' are constructed from the arcs in M m as follows: For each arc a in A m , • if arc a runs from X i(S ( C – 1 )+) to P S V ∈ PreVars(R) } in OCs ( if ( V ∈ OutVars(CE) ) then ( //found a match. Add the link and adjust OCs NewOCs = NewOCs ∪ { V p ∈ PreVars(CE) } OldOCs = OldOCs + V L = L + ( S i →S C ) V is a utility preconditon of S i } ∈ OCs ⎬ ⎩ ⎭ . The Discover_Links algorithm discovers a set of causal links, that in conjunction with the actions in Q and the ordering constraints S i < S i + 1, i ∈ { 1, …, n – 1 } specify a complete plan. Proof , therefore this plan is complete. in P R , then V = v is pertinent P in the P R . Proof is first added to the plan. per C, 0 is the set of values that are pertinent . The cost functions for the possible actions in A are all greater than zero. UDTPOP with the appropriate search control strategy can identify a plan P' that has the same topological sort Q . Proof be a series of planning domains with initial conditions given by IC k . Let S ICk be an initial conditions action with conditional effect IC k . We will show that UDTPOP can use the “planner execution trace” for the derivation of P k solving D k = in order to generate a plan for P k + 1 solving D k + 1 = . Thus UDTPOP with the appropriate search control can find the plan P n = P' solving D = D n = that has the topological sort { S i } ni = 1 . . Inductive Step: S IC . Pk in this graph, we will search backwards over the set of V in the exemplar plan P′ to find the latest action that provides effective support for . We know that there must be one step that provides this because, at the very least, the initial conditions contains only effective conditional outcomes. All of the later actions in the causal link chain that provide only passive support to will be linked via persist-support. This situation is illustrated in Figure 130. open). is not possible). In subset 2, P { V -, C } > 0 but P { u > u min V +, C, Δ 1, Δ 2 } = 0 ( is possible, but not pertinent). In subset 3, P { V -, C } > 0 and P { u V + } > 0 . . The cost functions for the possible actions in A are all greater than zero. DTPOP with the appropriate search control strategy can identify a plan P' that has the same expected value as Q . Proof be a series of planning domains with initial conditions given by IC k . Let S ICk be an initial conditions action with conditional effect IC k . We will show that DTPOP can use the “planner execution trace” for the derivation of P k solving D k = in order to generate a plan for P k + 1 solving D k + 1 = . Thus DTPOP with the appropriate search control can find the plan P n = P' solving D = D n = that has the topological sort { S i } ni = 1 .. This marked set represents the portions of the exemplar plan that can be identified using only the UDTPOP mechanisms. Mark the steps that contain these marked conditional effects and value functions. 310. Go to 1 and repeat. This procedure must mark all of the steps in the optimal plan. If the step is not marked, then it is possible to increase the value of the plan by eliding the step. Every arc in the plan can be added by the normal UDTPOP mechanisms except for the forward arcs. We can reconstruct all of the causal links in the plan as follows: 1. add all 2. add all 3. add all 4. add all 5. etc... . Inductive Step: S IC . Pk: An add-forward operation to add this link can be added after every add-backward operation that has an index less than i. This restriction ensures that there will be an open uncertainty available to serve as the root for this link. If the step has not been added yet, use add-step-forward to add this link. Otherwise, use add-link-forward. Case 2: Link L is labelled . We can add the causal link to establish L at any time after the time that the a precondition exists for L. Eventually one will be added, possibly after an add-forward operation adds all of the arcs of precedence i - 1. There are two possibilities for adding this link: • Say that an outcome variable of CE is protected by a link added either by add- 
