Learning of Tool Affordances for Autonomous Tool Manipulation *Raghvendra Jain ,Tetsunari Inamura (The Graduate University for Advance Studies SOKENDAI) Abstract— We present the concept of Tool Affordances to plan a strategy for target object manipulation by a tool via understanding of bi-directional association between Actions, Tools and Effects. Tool Affordances include the awareness within robot about the different kind of effects it can create in the environment using an action and a tool. Robot learns tool affordances by exploring the environment through its motor actions using different tools and learning their association with observed effects. The strength of our model is the robots ability of prediction and inference given some evidence. To deal with uncertainty, redundancy and irrelevant information Bayesian Network as the probabilistic model is chosen for implementation of our Tool Affordance model. We demonstrate a preliminary experiment where robot uses learnt Tool Affordances to correctly infer the most appropriate novel Action and Tool given the observed effects. I. INTRODUCTION Enabling robot to do everyday tasks with human competence requires intelligent use of tools that involves manipulation of target objects. It involves understanding the world well enough to know whether a change is needed and/or possible, and then forming a plan to use a known/unknown tool to implement that change, is very important. But at the same time it is an infinitely open challenge and demands to be addressed by the robotics researchers. But unfortunately the study of autonomous robotic tool manipulation has been rare Conventional researchers [1][2][3] pose either of the four following problems Problem1: Experience of learning of one tool cannot be applied to another tool [2][3] Problem2: The tools used in evaluation phase are minor modification of the one used in learning phase , thus evaluation criteria does not incorporate novelty of a tool [1] Problem 3: The bidirectional association between actions , tools and effects resulted due to target manipulation is not learned , hence robot cannot predict the effect given the cause and cannot infer the cause give effects [3] .Here the cause of manipulation is actions and tools and effect is the result of manipulation on target object. Thus an integrated solution to above mentioned problems is required to plan a manipulation strategy to solve similar task using novel tools and novel task using either similar or a novel tool . Our work attempts to solve above mentioned problems using the concept of Tool Affordances as explained in III. Raghvendra Jain and Tetsunari Inamura are with The Graduate University for Advanced Studies , Japan ; e-mail: [email protected] and [email protected] ).

II RELETAED WORK Stoytchev[1] proposed the concept of exploratory interaction with the environment to learn tool manipulation skill via learning Tool Affordances based in robot embodiment. Problem1 is addressed by learning tool affordances using random behavior babbling and solution validated during evaluation phase using tools similar to the ones used for learning; however evaluation using unfamiliar tools is not considered (Problem2). Another drawback is that sufficient focus on prediction and inference capability has not been given that leaving Problem3 largely unsolved. Stoytchev advanced his work[4][5] to find functional similarities between tools based on hierarchical representation of outcomes of tool manipulation but the notion of functional similarity is not yet incorporated in planning a strategy for tool manipulation. Kemp, Charles C et al. addressed visual tool tip detection and position estimation by using taskrelevant information autonomously extracted from robot’s sensor channels [2]. In their work the tip of the tool is taken as task relevant feature for tool manipulation and controlled throughout the task. The drawback is that task relevant features are limited to robot’s perceptual routine, and learning method restricts the choice of the tool and task. Nabeshima el al developed an adaptive body schema for robotic tool use based on visual feedback [3]. They proposed temporal integration of multisensory information as an underlying mechanism of robot learning to manipulate tools. The drawback of their work is that it does not incorporate the concept of shape of tools and target objects that have shown to influence body schema [3]. They later proposed a tool body assimilation model using which tool retrieves an unseen object towards the robot through a simulation model[6]. The work proved effective with a variety of rigid variety of tools with predefined features. These works [2][3][4] however do not enable the experience of learning of one tool to be applied to another tool (Problem1) , also [2][3] they did not address Problem3.Our proposed model incorporates the concept of shape of tools and relatively complex manipulation tasks are used for experiments. III .OUR CONCEPT OF TOOL AFFORDANCES Affordances encode the relationship between agent and its environment, thus depending on its motor and sensing capabilities[7]. Developmental robotics [8] suggests affordances as a higher level concept that an agent gains through its interaction with the environment [9]. The interactionist view of learning the coupling between perception , action and the environmental information is very well received in robotics community [1] for learning [10] , decision making [11] , navigation [12] as well as

extended to the problem of autonomous manipulation [1] .

robot tool

To solve the problems mentioned in I. authors present the concept of Tool Affordances. Our definition of Tool affordances includes the awareness within robot about the different kind of effects it can create in the environment using a tool. It incorporates the association of effects and abstract geometrical properties of tools with the perception of the initial environment and the executed action. Such an association can be learned by the robot through interaction with the environment using tool enabled exploratory behaviors. Acquisition of Tool Affordances can solve the problems mentioned in I in following manner. We propose novel solutions to four problems mentioned above. Solution to Problem1: Since all the tools differ on the basis of their functionalities e,g a knife can cut some fruit , and spoon can be used for serving some food , a hammer can be used to make an impact on some object . All these functionalities in turn depend on certain geometrical features of the tool e.g sharp edge or blade for knife and scissor small shallow bowl, oval or round, at the end of a handle of a spoon .Tool Affordances incorporate the geometrical features of the tool that help in defining the functionality of a tool. We call these geometrical features are called as functional parts of the tool. The notion of functional part is independent of its size, but depends on the abstract shape. Thus tools having similar functional parts can be used to generate similar effects. Thus if our learning method is based on the functional part of the tool, rather than the tools itself, it can generalize to a novel set of tools that have the same functional part. For example if robot can learn Tool Affordances for knife , where the functional part is its sharp edge , then all tools of different shapes with sharp edges can be used. Solution to Problem2: Learning of Tool Affordances enables robot to understand the effects of novel tools as the learning is based on functional parts not directly on tools itself. Thus robot do not require any additional learning for the novel tools that incorporates the functional part already used during learning phase. Solution to Problem3:

Fig1: Tool Affordances encode relationships between Actions, Tools and Effects.

Our implementation of Tool Affordances is based on Bayesian Networks (BN). BN offers a flexible way to store effects, tool information and actions so that association or dependencies between them can be learned. These learned associations can be used to make a query given some evidence. For example, given the Action and Tool robot can predict the effects and given the observed or desired effects robot can infer the cause of effects i.e. used Action and Tool .The table in Fig 1 shows the proposed use of Tool Affordances. The probabilistic semantics used in Bayesian Networks deals with uncertainties, redundancy and irrelevancy as well. Acquired skill using learnt Tool Affordances can be used for different set of tools as suggested in Solutions to Problem1 and Problem2 and solution to Problem3 suggests that probabilistic representation of dependencies within data, allows the robot to make probabilistic queries for prediction, inference and planning. Thus given some desired effect, robot can select the appropriate tool from a set of available tools and appropriate action from set of possible actions. Such capability allows robot to achieve the desired effect and solve a task using novel tools and actions. To represent effects as sequence of several state spaces learnt during Tool Affordances is subject to future work. IV IMPLEMENTATION A. Definitions Beck [13] who’s tool taxonomy is widely adopted today suggested the most comprehensive definition “Tool use is the external employment of an unattached environmental object to alter more efficiently the form, position, or condition of another object, another organism, or the user itself when the user holds or carries the tool during or just prior to use and is responsible for the proper and effective orientation of the tool “. Motor Primitive: An action unit that has a semantic meaning e.g grasp, approach, rotate wrist Action: Action: An action is the manipulation of the environment to change from one state to another state, where state is the instantaneous representation of observed environmental features. The observed features used in this paper are shown in Table 2. We define an action as superimposition and/or concatenation of motor primitives. e.g. approach-> contract arm->rotate wrist. Effect: Change of features ( refer Table 2) of the entities (object, tools etc ) within an environment after a time step t is called effect of that duration . It is represented as 12 dimensional vector using notations E and e

B. Formalization of Tool Affordances Our proposed framework incorporates the concept of developmental learning of robot based in robot embodiment. We propose formalization for learning tool affordance using relational instances similar to Mehmet at al [10]. The relational instance is of the form ( effect , ( state , tool-part , Action ) ) i.e. potential to generate some effect , through application of some Action using certain part of the tool given a certain environmental state. State refers to the features given in Table 2. One many argue that other features (shape, size etc) should also be a part of relational instance as it inadvertently influences the resulting effects and may change the degree of association or the association itself. But to keep our current work simple objects features are not included and is subject to consideration in future. Let a set of Actions A available to robot as A = {A1, A2 , A3 , A4 ……. Ai } Since all the task examples taken in our work are selected with the intention to show the robot acquiring skill of autonomous tool manipulation, the object always requires a tool to manipulate it . An action Ai is applied to an object o using tool T and produces effect e , where e ( refer Table 2) .Let a set of functional parts F available to the robot as F = {f1, f2 , f3 , f4 ……. fi } Each tool has atleast one functional part. For example, the L shaped tool has corner, horizontal and vertical bars. The T shaped tool has two corners in addition to L shaped too. Thus in order to manipulate an object we need a pair of action Ai and functional part of the tool fj. . A pair (Ai , fj. ) manipulates the object o and produces effect e ( Table 2) g(o , (Ai , fj. ) ) = e , where e , where g is the mapping function. There can be several such pairs available to the robot for manipulating a target object, and we call them manipulator pairs. Set of manipulator pair can be defined as Mp = { Mp1, Mp 2 , Mp 3 ……. Mp i } where Mp i (Aj , fk. ) where Aj A , fk F For some target object there may be a number of possible manipulator pairs that produce similar effect in a given situation .The manipulator pairs who upon acting on an object o bring the similar effect; they comprise the functionally equivalent set. A functionally equivalent set q can be defined as q = { Mp i } where g(o, Mp i ) e where e During exploration the agent assigns equal value to the elements of functionally equivalent set, but given some specific condition e.g force, torque, execution time etc robot may prioritize and chose one the suitable pair by assigning some weights to each pair. Acquiring the knowledge of functionally equivalent set is an important aspect of learning

tool affordances. As an alternative approach to selfexploration, such knowledge can be acquired by observing the teacher demonstration as well. To summarize, the proposed developmental process consists of three steps: 1. Interaction Step 1: Robot selects a tool and action randomly to collect the relational instances described by exploring the features given in Table1. The set of possible pairs of functional parts and Actions shown in table 1 are used to manipulate the target object and observing the effect. The exploration for one such manipulator pair is done several times to gather variety of data for making good generalization Exploratory Behaviors A1 :Contract Arm A2 :Slide Left A3:Pull-ArmDiagonally A4 :Slide Right A5: Extend Arm

Functional Part f1: Corner f2:Vertical f3:Horizontal

Manipulator pair used A1 , f3 A2 , f2 A3 , f1 A4 , f2 A5 , f3

Table 1Exploratory Behaviors used to manipulate target object in current work. Step 2: perceives and records the features given in Table2 before the start of tool manipulation and after the manipulation has ended . In simulation environment it is very straightforward to get the accurate observations. The aim of recorded information shown in Table 2 and used in the nodes of Bayesian Net of Fig 2 , is to associate observations with the functional part and Action Observation , ,

Meaning Position and orientation of, tool and target object

Table 2 : Recorded Observation at start and end of manipulation. 2. Learning Tool Affordances Step 3: derivation of generic affordances relations from the relational instances through learning the encoded probabilistic dependencies using Bayesian Network model 3. Evaluation of the Learned Affordances Step 4: The proposed evaluation of learned Tool Affordances is shown in the table of Fig1.In addition, we would like to 1. After observing the effects from self-exploration by the robot, infer and construct functionally equivalent set for a target object for different situations { Mp i } where h is inverse mapping function.

2. Measuring the similarity of outcome of object manipulation using a functionally equivalent pair in terms of effects g(o, Mp i ) e and e where g is mapping function While planning a task can be solved by breaking into several sub-tasks, with each sub-task having its own goal or say desired effect. Manipulation strategy to achieve individual desired effects cumulatively would lead to complete task solving strategy. Thus overall task solving strategy α can be decomposed into a temporal sequence of pairs of functional part and Action α = [ α[0] α[1] α[2]…… α[T] ] where α[t]= (Ai , fj ) kt kt=id of pair at time t And there can be several α’s for for the same task depending of the functionally equivalent pairs for a sub-task IV SYSTEM DESCRIPTION We chose Bayesian Network to store Actions and functional parts of the tool using discrete nodes as the representation of their effects. The observed multidimensional effects are represented implicitly as feature changes over time. Features are stored as continuous Gaussian nodes due to higher granularity. Feature values are recorded at start of the action (initial state) and after the object manipulation has ended (end state).Thus there are two rows; first row has nodes representing features at initial state and second row has nodes representing features at end state. Since the influence of one feature over other feature’s change is not clear, we decided to connect each node in the first row to the feature nodes in second row .The values of nodes representing feature vales over time is represented effects of that duration. Our implementation of Bayesian Networks is inspired from the previous works [8] [14].

effects as evaluation data and learned Tool Affordances using BN will infer the correct pair of functional part and Action. The manipulator pair for which highest likelihood is determined is considered as best classification for the evaluated sample. If two or more manipulator pairs have the same likelihood for an evaluated sample, they are grouped as functionally equivalent pairs. For inference the Junction Tree algorithm [16] has been used due to its feature of reducing complexity while computing the joint probability of nodes. The proposed strategy and algorithms are independent of robotic platforms. V. EXPERIMENTAL ENVIRONMENT A. Virtual Manipulator All experiments were performed using simulated virtual manipulator in simulator . Using virtual manipulator means that a random force is applied to the used tool and target object is manipulated. We chose to use virtual manipulator instead of some real manipulator arm as the focus is on learning of tool manipulation independent of robot embodiment. The learned strategy can later be adapted to different robot embodiments by using constraints based approach. B. Tools and target object The modeled tools incorporate the abstract geometrical properties that enable manipulation i.e functional parts. In total three functional parts are covered which are Corner , Vertical Bar and Horizontal Bar. The tool used was L stick as it contains all the functional parts. f2 f3

f1

Fig3 L to R :.1. Tshaped Tool 2. Stick 3. Inverse L shaped tool

Fig2 : Bayesian Net structure to learn Tool Affordances shown in Fig1 The BN’s are implemented using the open source .We use the Maximum Likelihood Parameter estimation [15] to adjust the weight parameters along the connection. For evaluation purposes we can simply present evaluation data to our network and given the evidence compare the likelihood of the data. In our current work, we provide the observed

VI. EXPERIMENTS AND RESULTS For each pair of Action and functional Part, a total of 160 times target is manipulated. In each trial of target manipulation, the position of target object is changed within the predefined workspace of the table and applied force is varied in next manipulation. The magnitude of force on virtual manipulator and area of workspace on table is

empirically chosen such that the target object does not fall from the table. The direction of force is defined according to the Actions given in Table 1. During the interaction phase , a total of 160 relational instances for each pair used in manipulation given by ( effect , ( state , functional Part, Action) ) are collected . A total of five manipulation pair shown in Table 1 are used. For each pair a total of 100 relational instances are used as training samples for Bayesian Network and 60 is kept for evaluation purposes. Thus in total there are 500 training samples for parameter learning and 300 evaluation samples. Tool Affordances are learned using the training data in the Bayes net given by structure shown in Fig 2.

We test its performance by giving the observed effects as input . The BN should be able to infer appropriate functional part and Action that might result in the observed and desired effects. To have such capability is very important for the robot to solve Problem 3 and Problem 4 which requires inference and planning. The evaluation data consisting of effects only for Sub-Plot1 and a combination of effects and functional part for Sub-plot2, Sub-plot2 and Sub-plot4. The marginal probabilities (likelihood) of the inferred element are calculated.

VII. DISCUSSION Four sub-plots are numbered from 1-4 in the right most of the fig5. The observed effects fed as input to the Bayesian Network shown in fig2 were the result of target manipulation using the manipulation pairs (refer Table1) collected over 60 trials for each pair. So in total 300 manipulation pairs are fed. Case1: The effects observed from manipulation of target object using A1 , f3 i.e Contract Arm with Horizontal Part . Case2: The effects observed from manipulation of target object using A2 , f2 i.e Slide Left with Vertical Part .

Case3: The effects observed from manipulation of target object using A3 , f1 i.e Pull-Arm Diagonally with Corner. Case4: The effects observed from manipulation of target object using A4 , f2 i.e Slide Right with Vertical Part . Case5: The effects observed from manipulation of target object using A5 , f3 i.e Extend Arm with Horizontal Part . Some evidence is given as input to the learnt Tool Affordance model and inference is made from that evidence.

The results obtained are shown are in Fig 5 . The data analyzed from results is shown in Table 3 and Table 4. Table 3 Evidence for BN Inference Sub-Plot Manipulator Pair SubPlot1 Effects f1 : Corner , Effects Action SubPlot2 f2:Vertical Part, Effects Action SubPlot3 f3 Horizontal Part , Effects Action SubPlot4 Table3: Association of Evidence , inference result and Subplot number of Fig 5 that shows the corresponding inference result . Table 4 Case1 Case2 Case3 Case4 Case5 Subplot1 A1 A2 A3 A4 A5 Subplot2 A3 A3 A3 A3 A3 Subplot3 A4 A2 A2 ,A4 A4 A2 ,A4 Subplot4 A1, A5 A5 A1, A5 A3 A3 Table3: Association of Sub-plot number of Fig 5 , cases and inference result corresponding to the cases and subplots. The results are shown in Table 4 so as to validate the inferring capability of Tool Affordances. For an ideal inference the probability should be at or near 1.0 in the appropriate regions of the data, and at or near zero elsewhere. If the marginal likelihood of inference i.e actions in Subplot1 (given functional part and effects as evidence) or manipulator pair in Subplot2 , Subplot3 and Subplot4 ( given only effects as evidence ) is near similar for an evaluation data, then those action and functional part put together are grouped in functionally equivalent set for that target object. As evident from Table 4 , for Subplot3 A2 ,A4 are elements of functionally equivalent set for Case 3 and Case 5 . Similarly , , for Subplot4 A1 ,A5 are elements of functionally equivalent set for Case 1 and Case 3 but looking at Fig 5 . But as it can be noticed in Fig 5 , that marginal likelihood for one action is considerably higher than the other action which is highlighted with bold fonts . Since target manipulation was performed in different conditions e.g position of object was varied in robot’s workspace , thus for some conditions functionally equivalent set can be used as lookup table for selection of an alternative manipulator pair when robot is unable to use the manipulator pair having higher likelihood due to some environmental constraints or physical limitation of the robot. VIII CONCLUSION We addressed the problem of planning a strategy to manipulate a target object using a tool . The three general problems not solved in conventional research has been presented and to solve them the concept of Tool Affordances has been proposed. The concept of Tool Affordances involves learning of bi-directional association between the actions, tools and effects through the target manipulation by random selection of Actions and Tools . Robot performs several manipulation trials to connect the data requires for such association. To generalize Tool Affordances the concept of functional part as abstract shape representation of functionality of tool is introduced and the formalization of

Tool Affordances is presented .The Tool Affordances are modeled using Bayesian networks that allow the robot to make probabilistic queries for inference, planning and prediction. To test the strength of learnt Tool Affordances an experiment is conducted where the effects only ,is given as input and robot is asked to infer the correct manipulator pair. In another experiment, given evidence is effects and functional part and robot has to select best Action for the situation .The notion of functionally equivalent set is also introduce for the manipulator pair that generate similar effects for a given situation .The experiments and results show that association has been learned successfully and robot is able to correctly infer the Actions that were not used in learning which can be used in planning a successful strategy. A richer set of experiments are planned to conduct so that a larger set of functional parts and Action can be incorporated in future. References [1] Stoytchev, A., "Learning the Affordances of Tools using a BehaviorGrounded Approach," In "Affordance-Based Robot Control," Springer Lecture Notes in Artificial Intelligence (LNAI) 4760, E. Rome et al. (Eds.), pp. 140-158, 2008. [2] Edsinger, Aaron and Kemp, Charles. "Manipulation in Human Environments", Proceedings of the IEEE/RSJ International Conference on Humanoid Robotics, 2006 [3] Nabeshima C ., Kuniyoshi Y.,and Lungarella M., “ Adaptive Body Schema for Robotic Tool Use “ , Advanced Robotics , Vol.20 , Nov 10 ,pp. 1105-1126(22),2006 [4] Sinapov, J. and Stoytchev, A., "Detecting the Functional Similarities Between Tools Using a Hierarchical Representation of Outcomes," In Proceedings of the 7th IEEE International Conference on Development and Learning (ICDL), Monterey, CA, Aug. 9-12, 2008 [5] Stoytchev, A., "Learning the Affordances of Tools using a BehaviorGrounded Approach," In "Affordance-Based Robot Control," Springer Lecture Notes in Artificial Intelligence (LNAI) 4760, E. Rome et al. (Eds.), pp. 140-158, 2008. [6] Cota Nabeshima, Yasuo Kuniyoshi and Max Lungarella :Towards a Model for Tool-Body Assimilation and Adaptive Tool-Use," Proceedings of The 6th IEEE International Conference on Development and Learning (ICDL-2007), London, United Kingdom, July, 2007 [7] J.J.Gibson , The Theory of Affordances . R.Shaw and J.Bransford , Eds. Lawrence Erlbaum 1977 [8] M. Lungarella, G. Metta, R. Pfeifer, and G. Sandini. Developmental robotics: a survey. Connection Science, 15(4):151–190, 2003. [9] P. Fitzpatrick, G. Metta, L. Natale, A. Rao, and G. Sandini. Learning about objects through action -initial steps towards artificial cognition. In Proc. of ICRA’03, pages 3140–3145, 2003. [10] Mehmet R Dogar , Maya Cakmak, Emre Ugur ,Erol Sahin , From Primitive Behaviors to Goal-Directed Behavior Using Affordances IEEERSJ International Conference on Intelligent Robots and Systems (2007) [11] L.Montesano , M.Lopes , A.Bernardino , and J.Santos-Victor .Learning Object Affordances : From sensory motor coordination to imitation , in Transactions on Robotics , 2007 [12] Dongshin Kim , Jie Sun , Sang Min , Oh James , M. Rehg , Aaron F. Bobick. Traversibility Classification Using Unsupervised On-line Visual Learning for Outdoor Robot Navigation. In Proc. of Int’l Conf. on Robotics and Automation (ICRA). 2006 [13] B. B. Beck, Animal Tool behavior: The use and manufacture of tools by animals. New York: Garland STMP Press, 1980. [14] M. Rudolph, M. Mühlig, M. Gienger, H.-J. Böhme, “Learning the Consequences of Actions: Representing Effects as Feature Changes”, in Proc. EST2010 [15] Christopher M. Bishop . Pattern Recognition and Machine Learning (Information Science and Statistics ) . Springer, 2007 [16] C. Huang and A.Darwiche . Inference in belief networks : Aprocedural guide . International Journal of Approximate Reasoning, 15(3) : 225263,1996