US006941290B2
(12)
United States Patent
(10) Patent N0.: US 6,941,290 B2 (45) Date of Patent: Sep. 6, 2005
Siskind
(54)
METHOD FOR COMPUTING ALL
OTHER PUBLICATIONS
OCCURRENCES OFA COMPOUND EVENT
,
FROM OCCURRENCES 0F PRIMITIVE EVENTS
.
.
.
. .
.
Kerridge et al; Synchronization Primitives for Highly Par allel Discrete Event Simulations; Proceedings of the 32nd Annual Hawaii International Conference on System Sci
(75) Inventor: Je?'rey Mark Siskind, Lawrenceville,
69685; V01- Tracl? 8; Jan- 5_8’ _1999; PP _1_10-* _
(73) Assignee; NEC Laboratories America, Inc”
Allen; Maintaining Knowledge About Temporal Intervals;
NJ (Us)
_
Siskind; Grounding Language in Percept1on;Art1?c1al Intel
l1gence Review; vol. 8; Dec. 1994; pp 371—391.* Princeton, NJ (Us)
Communications of the ACM; vol. 26, Iss. 1; Nov. 1983; pp 832—843.*
(*)
Notice:
Subject to any disclaimer, the term of this
ChOW; A GeneraliZed Assertion Language; Proceedings of
patent is extended or adjusted under 35
the 2nd International Conference on Software Engineering;
U.S.C. 154(b) by 635 days.
OCI- 1976* Thiele et al; On FuZZy Temporal Logic; Second IEEE International Conference on FuZZy Systems; vol. 2; Mar. 28—Apr. 1, 1993; pp 1027—1032.* Abe. N. et al., “A Plot Understanding System on Reference
(21) APPL NO: 09/916,249 (22)
Filedi
Jul- 30, 2001
(65)
Pnor Pubhcatlon Data US 2002/0138458 A1 Sep. 26, 2002
.
.
.
to Both Image and Lanaguge,” Proceedings of the Seventh
International Joint Conference on Arti?cial Intelligence, Vancouver, Canada, pp. 77—84, Aug. 1981.
Related US. Application Data (60) 2P(r)(())‘6isional application No. 60/247,474, ?led on Nov. 10,
(Continued) Primary Examiner_AnthOny Knight Assistant Examiner—Meltin Bell
(51)
Int. c1.7 ......................... .. G06F 17/00; G06N 7/00;
G06N 7/08
(57)
ABSTRACT
A method for computing all occurrences of a compound event from occurrences of primitive events where the com
U-Sn Cl-
(58)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..
pound events. event The method is a de?ned includes Combination the steps of of: the (a)
Field of Search ............................ .. 706/25, 45, 58,
706/49, 53, 56, 57; 704/251 (56)
primitive event type is true; and (d) computing the com pound event occurrences, such occurrences being speci?ed
U.S. PATENT DOCUMENTS 3,647,978 A
*
* 10/1992
3/1972
5,301,320 A
*
5,966,523
A
*
6,021,403 A
*
Hill .......................... .. 704/251 Goodridge .... ..
.. 704/219
4/1994 McAtee et al. . 10/1999
Uchino
. ... ... .
2/2000 HoWitZ et al.
primitive event types; (b) de?ning combinations of the
primitive event types as a compound event type; (c) input ting the primitive event occurrences, such occurrences being speci?ed as the set of temporal intervals over which a given
References Cited
5,153,922 A
de?ning
705/9 . . . ..
703/2
706/45
as the set of temporal intervals over which the compound event type is true, where the sets of temporal intervals in steps (c) and (d) are speci?ed as smaller sets of spanning intervals, each spanning interval representing a set of inter vals.
6,424,370 B1 * 7/2002 Courtney ...... .. 348/143 6,785,663 B2 * 8/2004 Wang et al. ..... .. 706/45 6,813,312 B2 * 11/2004 Tullberg et al. ..... .. 375/240.01
5 Claims, 19 Drawing Sheets
De?ne pnmitive event type!
De?ne mrnbinniionr of iii: primitive event types as a
compound ever-ii type
Input the primitive event uccurrenccl. such occurrences
106
being speci?ed in the Sci pr
\empoml mbtrVI-l! werwhich a g'ven primitive event type is true, wherein the set: of
temporal intervals are speci?ed B5 smaller sels ufspanning
intervals, mn spanning interval representing a set ofinlervlls
Compute the compound :venr occurrences, such occurrences being rpeci?ed as me set of lempnru]
iniewnir iwer which the compound event type is true, wherein iii: sets
cftempm'd-l inmvnis m speci?ed as n-niiier rm ufspanning intervals,
mir spinning interval representing a set of intervals
103
US 6,941,290 B2 Page 2
OTHER PUBLICATIONS
Abe, N. et al., “A Learning of Object Structures by Verbal ism,” COLING 82, pp. 1—8, 1982. Adler, M.R., “Computer Interpretation of Peanuts Car toons,” 5th International Joint Conference on Arti?cial Intel
ligence, Cambridge, MA, pp. 608, Aug. 1977. Allen, J .R., “Maintaining Knowledge About Temporal Inter vals,” Communications of the ACM, vol. 26, No. 11, pp. 832—843, Nov. 1983. Blum, M. et al., “A Stability Test for Con?gurations of Blocks,” Arti?cial Intelligence Memo No. 168, Massachu setts Institute of Technology, Feb. 1970.
Bobick, A.F. et al., “Action Recognition using Probabilistic Parsing,” Proceedings of the IEEE Computer Society Con ference on Computer Vision and Pattern Recognition, pp. 196—202, Jun. 1998.
Borchardt, G.C., “A Computer Model for the Representation and Identi?cation of Physical Events,” Masters Thesis, University of Kansas, May 1984. Borchardt, G.C., “Events Calculus,” Proceedings of the Ninth International Joint Conference on Arti?cial Intelli
gence, pp. 524—527, Aug. 1985. Brand, M. et al., “Sensible Scenes: Visual Understanding of
Siskind, J.M., “Unsupervised Learning of Visually—Ob served Events,” AAAI Fall Symposium Series on Learning
Complex Behaviors in Adaptive Intelligence Systems, pp. 82—83, 1996.
Siskind, J .M., “Visual Event Perception”, Proceedings of the 9th NEC Research Symposium, Princeton, NJ, Mar. 1999. Siskind, J .M., “Visual Event Classi?cation via Force Dynamics,” Proceedings of the Seventeenth National Con ference on Arti?cial Intelligence, Aug. 2000. Siskind, J .M. et al., “A Maximum—Likelihood Approach to Visual Event Classi?cation,” Proceedings of the 4th Euro pean Conference on Computer Vision, Cambridge, UK, pp.
347—360, Apr. 1996. Starner, T.E., “Visual Recognition of American Sign Lan guage Using Hidden Markov Models,” Masters Thesis, Massachusetts Institute of Technology, Feb. 1995.
Talmy, L., “Force Dynamics in Language and Cognition,” Cognitive Science, vol. 12, pp. 49—100, 1988. Thibadeau, R., “Arti?cial Perception of Actions,” Cognitive Science, vol. 10, No. 2, pp. 117—149, 1986.
of the Eleventh National Conference on Arti?cial Intelli
Tsuji, S. et al., “Understanding a Simple Cartoon Film by a Computer Vision System,” Proceedings of the 5th Interna tional Joint Conference on Arti?cial Intelligence, Cambridge
gence, pp. 588—593, 1993.
MA, pp. 609—610, Aug. 1977.
Fahlman, SE, “A Planning System for Robot Construction Tasks,” Arti?cial Intelligence, vol. 5, No. 1, pp. 1—49, 1974.
Tsuji, S. et al., “Three Dimensional Movement Analysis of Dynamic Line Images,” Proceedings of the Sixth Interna
Complex Structures Through Causal Analysis,” Proceedings
Krifka, M., “Thematic Relations as Links BetWeen Nominal
Reference and Temporal Constitution,” Lexical Matters, Sag, I.A. (eds.), pp. 29—53, 1992. Mann, R. et al., “ToWards the Computational Perception on
Action,” Proceedings of the IEEE Computer Society Con ference on Computer Vision and Pattern Recognition, Santa
Barbara, CA, pp. 794—799, 1998. Mann, R. et al., “The Computational Perception of Scene
tional Joint Conference on Arti?cial Intelligence, Tokyo,
Japan, pp. 896—901, Aug. 1979. Tsuji, S. et al., “Tracking and Segmentation of Moving Objects in Dynamic Line Images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 2, No. 6, pp. 516—522, 1980. WaltZ, D.L., “ToWard a Detailed Model of Processing for
Dynamics,” Computer Vision and Image Understanding,
Language Describing the Physical World,” Proceedings of
vol. 65, No. 2, pp. 113—128, Feb. 1997.
the Seventh International Joint Conference on Arti?cial
McCarthy, J., “Circumscription—A Form of Non—Mono tonic Reasoning,” Arti?cial Intelligence, vol. 13, pp. 27—39,
Intelligence, Vancouver, Canada, pp. 1—6, Aug. 1981. WaltZ, D.L., “Visual Analog Representations for Natural Language Understanding,” Proceedings of the Sixth Inter
1980.
Okada, N., “SUPP: Understanding Moving Picture Patterns Based on Linguistic Knowledge,” Proceedings of the Sixth International Joint Conference on Arti?cial Intelligence,
Tokyo, Japan, pp. 690—692, Aug. 1979. Regier, T.P., “The Acquisition of Lexical Seminatics for Spatial Terms: A Connectionist Model of Perceptual Cat egorization,” Ph.D. Thesis, University of California, Berke ley, 1992. Shoham, Y., “Temporal Logics in A1: Semantical and Onto logical Considerations,” Arti?cial Intelligence, vol. 33, pp. 89—104, 1987. Siskind, J.M., “Naive Physics, Event Perception, Lexical Semanics, and Language Acquisition,” Ph.D. Thesis, Mas sachusetts Institute of Technology, 1992. Siskind, J.M., “Axiomatic Support for Event Perception,” Proceedings of the AAAI—94 Workshop on the Integration of Natural Language and Vision Processing. Seattle, WA, pp. 153—160, Aug. 1994.
national Joint Conference on Arti?cial Intelligence, Tokyo,
Japan, pp. 926—934, Aug. 1979. Yamato, J. et al., “Recognizing Human Action in Time—Se
quential Images using Hidden Markov Model,” Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pp. 379—385, 1992. ChoW; A Generalized Assertion Language; Proceedings of the 2nd International Conference on SoftWare Engineering; Oct. 1976; pp 392—399.*
Thiele et al; On FuZZy Temporal Logic; Second IEEE International Conference on FuZZy Systems’, vol. 2., Mar.
28—Apr 1, 1993., pp 1027—1032.*
Kerridge et al; SynchroniZation Primitives for Highly Par allel Discrete Event Simulations’, Proceedings of the 32nd Annual HaWaii International Conference on System Sci
ences, vol. Track 8, Jan. 5—8, 1999, pp 1—10.*
Siskind, J .M., “Grounding Language in Perception,” Arti? cial Intelligence RevieW, vol. 8, pp. 371—391, Dec. 1994.
* cited by examiner
U.S. Patent
Sep. 6,2005
Sheet 1 0f 19
US 6,941,290 B2
102
De?ne primitive event types
De?ne combinations of the primitive event types as a
104
compound event type
Input the primitive event occurrences, such occurrences
106
being speci?ed as the set of
temporal intervals over which
/
a given primitive event type is true, wherein the sets of
temporal intervals are speci?ed as smaller sets of spanning
intervals, each spanning interval representing a set of intervals
Compute the compound event
10
occurrences, such occurrences being / speci?ed as the set of temporal intervals over which the compound event type is true, wherein the sets
of temporal intervals are speci?ed as smaller sets of spanning intervals,
each spanning interval representing a set of intervals
FIG. 1
8
U.S. Patent
Sep. 6, 2005
Sheet 2 0f 19
US 6,941,290 B2
Start
l Label primitive subexpressions of (D with spanning intervals that represent the sets of intervals over which the
corresponding primitive event types hold
Are there any
subexpressions that have not been labeled
with sets of spanning intervals?
Stop: output the set of spanning intervals that label the whole
(root) expression CD
Let (1)’ be some subexpression of (I) such that CD’is not labeled with sets of
spanning intervals 'but for vyhich all Y subexpressions (DI , MCI)" of II)’ are labeled with sets of spanning intervals
FIG. 2
Apply the appropriate formula for e (M, ') using the subroutines (i), i1 (1 i2,-1i, sPAN(i1, i2),€D(r, i), and $J(i,r, j) to compute a set of
spanning intervals to label (1)’
U.S. Patent
Sep. 6, 2005
US 6,941,290 B2
Sheet 5 0f 19
UENMUNtQHE 25ENNQBEEE32
uNNamkm
.65
mm@E
U.S. Patent
Sep. 6, 2005
Sheet 6 0f 19
US 6,941,290 B2
(PICK-UP "ovum Rm GREEIONHOJ]. (14121)} ‘
(PUT-DOV)! MOVING RED GREEN)“ [(0.14] . [23 .321"
(SUPPORTED? mmuomm
(suwmu'su'! novwm?llmza] )} (survmmm'! nsmuumazn) (swmn'rs? MOVING nzmuuomn) (suvvoa'rsv luzn mvmm? [[14:231» (SUPPORTS? mm novmcnflillnzah) (SUPPORTS? cam: amn?lmazln
(surwunrznv novmcnuunsl). [[24:2s])) (swrom'sr am nuvlncnunual), (lumen) (swroa'rs'r nuvnm nmn?nanzl )1 (SUPPORTS? GREEN REDMHIOIND}
(CONTACTS? an) caszunuumszn)
(ATTACHED? sumac am)“ [[0:23] >) (ATTAGKED'! RED cmlln?lmnln
FIG. 6A
FIG. 6B
U.S. Patent
Sep. 6,2005
Sheet 7 0f 19
US 6,941,290 B2
. M
U.S. Patent
US 6,941,290 B2
BEEmm3“295@N069a
U.S. Patent
Sep. 6,2005
Sheet 9 0f 19
US 6,941,290 B2
m
Spa wa
Ow.DE 25¢3
,9, “w W .. ,
U.S. Patent
Sep. 6,2005
Sheet 10 0f 19
US 6,941,290 B2
0Kb5 »
0258m
.05Dw
U.S. Patent
Sep. 6,2005
Sheet 11 0f 19
US 6,941,290 B2
GEmi.
U.S. Patent
Sep. 6,2005
Sheet 12 0f 19
(arr-nova nuvm m nwsn?lmuLluwll) (STACK WWII; RED ILUE GEDOO‘IHOJil-[ZIJOIH
FIG. 8A
FIG. 8B
FIG. 8C
FIG 8D
FIG. 8B
US 6,941,290 B2
U.S. Patent
Sep. 6,2005
Sheet 14 0f 19
US 6,941,290 B2
U.S. Patent
Sep. 6,2005
#N$35E
Sheet 15 0f 19
US 6,941,290 B2
U.S. Patent
Sep. 6,2005
Sheet 16 0f 19
US 6,941,290 B2
Q6 6
E
U.S. Patent
Sep. 6,2005
Sheet 17 0f 19
US 6,941,290 B2
FIG. 10A
FIG. 10B
FIG. 10C
FIG. 10D
U.S. Patent
Sep. 6,2005
Sheet 18 0f 19
US 6,941,290 B2
UEMH
25¢2
22:8.“
95EM:
.05m: N~Deni
