Vector Symbolic Architectures Answer Jackendoff’s Challenges for Cognitive Neuroscience Ross Gayler [email protected]

15/10/2004

Ross Gayler, RNI seminar, Menlo Park

1

Overview • Jackendoff’s Challenges • Vector Symbolic Architectures • Challenges & Responses

15/10/2004

Ross Gayler, RNI seminar, Menlo Park

2

Origin of Jackendoff’s Challenges • Language is a mental phenomenon and neurally implemented • Cognitive neuroscience view of linguistic phenomena seems naive • Identifies core linguistic functionality:

– Not provided by or obviously inherent in current connectionist models (Marcus, 2001) – Arguably core cognitive functionality

15/10/2004

Ross Gayler, RNI seminar, Menlo Park

3

Vector Symbolic Architectures • A family of connectionist approaches

– From Smolensky’s Tensor Binding Networks – Little-known and little-used • Gayler, Kanerva, Plate, Rachkovskij

• Provide ‘symbolic’-like functionality

– Represent & manipulate recursive structures

– Without training (because of architecture) – Practical basis for implementation 15/10/2004

Ross Gayler, RNI seminar, Menlo Park

4

VSA Architectural Commitments • Network of connectionist units

– Single scalar output, multiple scalar inputs, fixed local numerical calculation

• High-dimensional vectors of unit outputs – 1,000s to 10,000s of units

• Distributed representations on vectors – Representations appear random

• Algebra-like operations on vectors – Statistical rather than exact

15/10/2004

Ross Gayler, RNI seminar, Menlo Park

5

Vector Operators • Vector operators vary by VSA • MAP Coding (Gayler, 1998) • Multiply (elementwise multiplication) – Binding (structural composition)

• Add (elementwise addition)

– Superposition (set composition)

• Permute

– Quotation (protection)

15/10/2004

Ross Gayler, RNI seminar, Menlo Park

6

Levels of Description Vector Representation Representational Architecture Cognitive Architecture

15/10/2004

VSA

Computer

High-dimensional vectors & Vector operators:  ,  , P( )

Binary words & Logical operators: AND , XOR , …

Fixed network of operators

CPU

Mapping of task onto lower levels

Program

Ross Gayler, RNI seminar, Menlo Park

7

Approach Taken • Accept the challenges at face value • Skate over VSA details • Don’t compare VSA with other connectionist approaches

15/10/2004

Ross Gayler, RNI seminar, Menlo Park

8

C1: Massiveness of Binding • Representations must be composed • A commonly used example of binding: – “red square” & “blue circle”

• Jackendoff’s example sentence:

– “The little star’s beside a big star”

– Encoded with approximately 130 tokens and 160 relations between tokens

• Must be rapid and cope with novelty 15/10/2004

Ross Gayler, RNI seminar, Menlo Park

9

The little star’s beside a big star

© Ray Jackendoff 2002; reproduced by permission of Oxford University Press 15/10/2004

Ross Gayler, RNI seminar, Menlo Park

10

R1: Massiveness of Binding (1) • VSAs do binding

– as an architectural primitive – as an untrained operation

– blind to the identity of the vectors

• Fast • Oblivious to novelty • Vectors being bound may be composites 15/10/2004

Ross Gayler, RNI seminar, Menlo Park

11

R1: Massiveness of Binding (2) • bind(a,b) = a*b

• unbind(cue,trace) = bind(inverse(cue),trace)

• Able to deal with noisy cues and traces • In MAP and Kanerva’s Spatter Coding – each vector is its own binding inverse – bind(a,a) = a*a = 1

– unbind(cue,trace) = bind(cue,trace)

– removes the need for an inverse operator 15/10/2004

Ross Gayler, RNI seminar, Menlo Park

12

R1: Massiveness of Binding (3) bind

unbind

a

a* b

b

b

bind a b 15/10/2004

a*b*b = a*(b*b) = a* 1 =a

p*q + r*s + ... a* b

unbind

a*b + ... b Ross Gayler, RNI seminar, Menlo Park

b*(a*b + …) = a*b*b + b*(…) = a + b*(p*q + ...) = a + (b*p*q + …) = a + noise 13

C2: The Problem of 2 • How are multiple instances represented? – “little star” & “big star”

• This is a problem for ‘localist’ connectionist representations

15/10/2004

Ross Gayler, RNI seminar, Menlo Park

14

R2: The Problem of 2 (1) • VSAs do superposition of entities • Only values exist in a given vector space star + star = 2 star

• Distinct entities must have distinct values • Need a representation such that any difference creates unrelated values frame1 = {is-a:star size:little} frame2 = {is-a:star size:big} similarity(frame1,frame2) = 0 15/10/2004

Ross Gayler, RNI seminar, Menlo Park

15

R2: The Problem of 2 (2) • Naive frame encodings don’t work (little*star) + (big*star) = (little + big)*star (is-a*star + size*little + colour*red) + (is-a*star + size*big + colour*blue) = (is-a*star + size*little + colour*blue) + (is-a*star + size*big + colour*red) 15/10/2004

Ross Gayler, RNI seminar, Menlo Park

16

R2: The Problem of 2 (3) • Frames represented as OAV triples • Frame ID is permuted frame contents P(is-a*star + size*little)*(is-a*star + size*little) = P(is-a*star + size*little)*is-a*star + P(is-a*star + size*little)*size*little

• Cross-binds all role:filler pairs • Easy to implement

P(a)*a

a P( ) 15/10/2004

Ross Gayler, RNI seminar, Menlo Park

17

C3: The Problem of Variables • Construe grammar ‘Rules’ as templates with variables • Productivity arising from variables • Typed variables take constrained values • How are typed variables implemented?

15/10/2004

Ross Gayler, RNI seminar, Menlo Park

18

R3: The Problem of Variables (1) • Variables as placeholders (Smolensky) • Bind a vector representing the identity of the variable with a value vector – variable_id*value

• Value can be retrieved from the binding • Variable_id and Value are just vectors – Could atomic or composite

– Variable and Value are interchangeable 15/10/2004

Ross Gayler, RNI seminar, Menlo Park

19

R3: The Problem of Variables (2) • Variables as targets for substitution • Bind a vector representing a substitution with a vector representing a structure – Substitute x for a: (x*a)

– Apply the substitution to (a*b) (x*a)*(a*b) = x*a*a*b = x*1*b = (x*b)

• Under substitution every component of a structure acts as a variable 15/10/2004

Ross Gayler, RNI seminar, Menlo Park

20

R3: The Problem of Variables (3) • Types are constraints on variables • Lexicalised grammars shift complexity from the rules to the atomic structures • Types and other constraints can be realised as additional attributes on VSA structures • Substitutions apply systematically across all the superposed structures in a space • Suggests constraint-based formalisms/DOP 15/10/2004

Ross Gayler, RNI seminar, Menlo Park

21

C4: Working Memory & LTM • Linguistic tasks require functional equivalence of Working Memory & LTM

– Different physical implementations in typical connectionist models argue against functional equivalence – Speed of learning

• Iterative learning (e.g. backpropagation) is implausibly slow

15/10/2004

Ross Gayler, RNI seminar, Menlo Park

22

R4: Working Memory & LTM (1) • Problem cast as physical implementation (activations vs synaptic weights) • Recast in terms of logical representations • MLP: WM vectors vs LTM matrices – WM and LTM items exist in different incommensurable spaces

• VSA: WM & LTM items in same space items to store 15/10/2004



cue trace

Ross Gayler, RNI seminar, Menlo Park

retrieved items 23

R4: Working Memory & LTM (2) • VSA WM/LTM distinction comes from persistence, not representational space • WM/LTM distinction reflected in patterns of interaction – WM interacts directly with WM

– WM interacts directly with LTM

– LTM with LTM only via items in WM

15/10/2004

Ross Gayler, RNI seminar, Menlo Park

24

R4: Working Memory & LTM (3) • Speed of learning in VSAs

– Binding in WM occurs in one pass

– Adding an item to a superposition in WM takes one pass

– Adding an item to a superposition in LTM takes one pass (logically)

• May take longer (physically) depending on method for implementing persistence

15/10/2004

Ross Gayler, RNI seminar, Menlo Park

25

Conclusion • VSAs meet Jackendoff’s challenges • Functionality comes from the algebraic properties of the representations – No training required

• Simple connectionist implementation • Harder to work with because of dependence on design instead of training 15/10/2004

Ross Gayler, RNI seminar, Menlo Park

26