A Complete Variational Tracker Ryan Turner, Steven Bottone, and Bhargav Avasarala Northrop Grumman Corporation

Full model: combine the assignment process and track models

– filtering + framing constraints + track management

k=1

P (Ak |Sk ) ·

NT Y

NT Y Y Akij k Ak0j p(zj,k |xi,k , Aij = 1) p(xi,k |xi,k−1)P (si,k |si,k−1)· p0(zj,k ) NZ (k)

i=1

j=1

i=1

Goal: compute p(Xk |Z1:k ), but “combinatorial explosion” in summing out A1:k , =⇒ ˆ k :k } for a sliding window w = k2 − k1 + 1 MHT with P (Ak1:k2 |Z1:k ) ≈ I{Ak1:k2 = A 1 2

VB Convergence (Radar)

2

Conjugate Assignment Prior (CAP)

10

−5

10

−10

10

−10

0

Tracking: • At each frame k, observe NZ (k) ∈ N0 measurements from real targets and clutter • Data association: infer assignment matrix A, Aij = 1 ⇔ track i assoc. to meas. j • Constraint: Each track is associated with at most one measurement, and vice-versa PNT PNZ • Mutual exclusion constraint: i=0 Aij = 1, j=0 Aij = 1, A00 = 0 • “Dummy row” and “dummy column” to represent clutter and missed detections • N D MHT: find MAP A for sliding window of last N − 1 frames

Assignment Matrices A1

X3

Z2

Full model joint: Sk :=

…

Z3 NT {si,k }i=1,

Xk Zk

Xk :=

ͲͲ ͳͲ

݂ேோ೅

NT {xi,k }i=1,

Ͳ

ௌ ݂ଵ଴

݂ேௌ೅ ଴

Zk =

Ͳͳ ͳͳ

ͳ

ௌ ݂଴ଵ ௌ ݂ଵଵ

݂ேௌ೅ ଵ

∙∙∙

∙∙∙

݂ே஼ೋ

Ͳ ͳ

∙∙∙

X2

Ak

݂ଵோ

ௌ ݂଴଴

݂ଵ஼

∙∙∙

Measurements Z1

A3

Sk

∙∙∙

(all) Track States X1

A2

…

30

0

5

10 15 LBP iteration

∙∙∙



ௌ ݂଴ே ೋ

20

ௌ ݂ଵே ೋ

݂ேௌ೅ ேೋ

NZ (k) {zj,k }j=1 .

Prior on assignment matrices: • Implicitly used in MAP literature (e.g. MHT) • Number of tracks NT is assumed known a priori and NZ is random • Bernoulli missed detections (PD ), Poisson clutter (Nc), meas. in arbitrary order QNT Nc =⇒ P (A|PD ) = λ exp(−λ)/NZ ! i=1 PD (i)di (1 − PD (i))1−di Track model: QK QK • Use a state space model: p(z1:K , x1:K ) = p(x1) k=2 p(xk |xk−1) k=1 p(zk |xk ) • Focus on LDS case: p(xk |xk−1) = N (xk |Fxk−1, Q), p(zk |xk ) = N (zk |Hxk , R) • Track meta-states: address track management; two-state Markov model with an active/dormant meta-state sk : PD when dormant ≪ PD when active

Variational Formulation

track 3

track 3

track 3

track 1

track 2

track 2

track 1

track 2

track 1

Easting

• Measure performance with SIAP and Rand index metrics

• VB tracker gets the scenario almost perfect, sets track to dormant when wrong • Poor no clutter (NC) ARI of OMGP due to lack of framing constraints 1 100 0.8 80 60 40

ij

0.6 0.4

20

0.2

0

0

PA

VB−DP

• Modify variational lower bound L to obtain a tractable algo. Factor graph for CAP: QNT R QNZ C Q NT Q N Z S – CAP(A|χ) ∝ i=1 fi (Ai·) j=1 fj (A·j ) i=0 j=0 fij (Aij ) P NZ P NT R C – fi (v) := I{ j=0 vj = 1}, fj (v) := I{ i=0 vi = 1}, fijS (v) := exp(χij v) • Reparameterize into binary factor graph, get Bethe entropy P NT P NZ P NT P NZ – Hβ [q(A)] = i=1 H[q(Ai·)] + j=1 H[q(A·j )] − i=1 j=1 H[q(Aij )] • Replaces the entropy H[q(Ak )] with Hβ [q(Ak )] in VB lower bound (L → Lβ ) • Update for q(X1:K , S1:K ) unchanged • Results in Bethe free energy objective for LBP when updating q(Ak ) Loopy BP: Define row and column messages C R C R C R := msg , ν , ν := msg , µ := msg µR Aij →fi fj →Aij fi →Aij ij := msgAij →fjC ij ij ij R R (0)/µ := µ Apply BP rules, simplify, and reparam. (˜ νijR := νijR(1)/νijR(0) and µ˜ R ij (1)): ij ij P NT C PNZ R exp(χij ) exp(χij ) C C C R R R µ˜ ij = l=0 ν˜il − ν˜ij , ν˜ij = µ˜C , µ˜ ij = l=0 ν˜lj − ν˜ij , ν˜ij = µ˜R C − log µ ˜ Get final result with: E[Aij ] = P (Aij = 1) = σ(χij − log µ˜ R ij ). ij

Truth OMGP

• 3D MHT better, but misses western portion of track 2 and swaps track 1 and 2

Efficient Assignment Matrix Update

ij

Truth MHT 2D MHT 3D

• OMGP and 2D MHT miss real tracks and create spurious tracks from clutter

• Exact inference on the full model is intractable • Use factorization constraint q(A1:K , X1:K , S1:K ) = q(A1:K )q(X1:K , S1:K ) QK QNT • Induced factorization: q(A1:K ) = k=1 q(Ak ) , q(X1:K , S1:K ) = i=1 q(xi,·)q(si,·) – State update: Kalman smoother with pseudo-meas. z˜i,k and meas. cov. R/E[di,k ] PNZ P NZ 1 k k – Detection prob.: E[di,k ] = 1−E[Ai0] = j=1 E[Aij ], z˜i,k := E[di,k] j=1 E[Akij ]zj,k • Meta-state update: use HMM forward-backward with emission log likelihoods ℓi,k – ℓi,k (s) := E[di,k ] log(PD (s)) + (1 − E[di,k ]) log(1 − PD (s)) , s ∈ 1:NS QK • Assignment update: q(A1:K ) = k=1 CAP(Ak |Eq(Xk )[Lk ] + Eq(Sk )[χk ]) • But, Eq(Ak )[Ak ] is intractable if q(Ak ) is CAP =⇒ modify inference with LBP

4

Truth VB

Performance

3

Model Setup

S3

10 20 VB iteration

S

C

VB

ARI

MHT 3D

NC−ARI

MHT 2D

0−1

OMGP

1 0.8 Performance

Easting

S2

0

−10

10

• Compare with standard methods: 2D and 3D (i.e. multi-frame) MHT trackers

Performance (%)

Northing

track 3 (Cessna)

track 1 (747)

S1

10

10 20 VB iteration

Northing

To posterior P (A|Z) what is theQconjugate prior to full likelihood p(Z, X|A)? QNcompute Q Q NT NT NT Z A0j Aij ⊤ p (z ) p(z |x , A = 1) p(x ) = p(x ) exp(1 (A ⊙ L)1) j i ij i i j=1 0 j i=1 i=1 i=1 Lij := log p(zj |xi, Aij = 1) , Li0 := 0 , L0j :=Q log p0(zj ) T =⇒ EF quantities base measure h(Z, X) = N i=1 p(xi), partition function g(A) = 1, natural parameters η(A) = vec A, and sufficient stats. T (Z, X) = vec L. =⇒ CAP P (A|χ): CAP(A|χ) := Z(χ)−1I{A ∈ A} exp(1⊤(χ ⊙ A)1) • Recover original prior: χij = σ −1(PD (i)) − log λ , χ0j = χi0 = 0 QNT −1 • Here, Z(χ) = Poisson(NZ |λ) i=1(1 − PD (i)). Computing E[A] or Z(χ) in general remains difficult =⇒ invoke LBP. General problem similar to permanent.

clutter (birds)

Meta-states

−5

10

−5

10

• Radar tracking example of L´azaro et. al. (2012) + clutter λ = 8, and PD = 0.5

track 2 (777)

1

0

10

0

10

Radar example:

• Develop process to handle track management in a model-based way

Track Swap

LBP Subroutine (Soccer)

10 0

• Get induced factorization: VB scales linear in window length, MAP is exponential • New approx. inference (LBP) for assignment and a conjugate assignment prior (CAP)

VB Convergence (Soccer)

5

Bethe energy (nats)

• First deterministic efficient approximate inference algo. for full tracking problem

k=1 K Y

Results

p(Zk |Xk , Ak )p(Xk |Xk−1)P (Sk |Sk−1)P (Ak |Sk ) = LB change (nats)

Introduce probabilistic tracking algorithm with mutual exclusion constraints and track management using variational Bayes (VB) and loopy belief propagation (LBP). Contributions:

p(Z1:K , X1:K , A1:K , S1:K ) =

K Y

5

LB change (nats)

Abstract

0.6 0.4 0.2 0

ARI

NC−ARI

0−1

Soccer data: • Preprocess video: multi-scale HOG features and sliding window SVM • Train params. with VB LB Lβ on first 1000 frames; Test set: 70 seqs. of 20 frames • Evaluate batch accuracy of assigning boxes to correct players • VB increases NC-ARI and lowers 0-1 clutter loss; VB-DP further lowers 0-1 loss