2014 Joint IEEE International Conferences on Development and Learning and Epigenetic Robotics (ICDL-Epirob) October 13-16, 2014. Palazzo Ducale, Genoa, Italy

Special Session: Dynamic Interactions Between Visual Experiences, Actions and Word Learning Beata J. Grzyb, Allegra Cattani, Angelo Cangelosi and Caroline Floccia Plymouth University Plymouth, United Kingdom [email protected]

Hanako Yoshida, Joseph Burling University of Houston, Houston, United States [email protected]

Swapnaa Jayaraman and Linda B. Smith Indiana University Bloomington, United States [email protected]

Alfredo F. Pereira, Isabel C. Lisboa, Emanuel Sousa, Jorge A. Santos, Wolfram Erlhagen, and Estela Bicho University of Minho Braga, Portugal [email protected]

Abstract—The primary aim of this special session is to inform the conference’s interdisciplinary audience about the state-of-the-art in developmental studies of action and language interactions. Action and language develop in parallel, impacting each other, and as such, bootstrap action, social, and cognitive development. We will present recent empirical evidence on developmental dependencies between visual experiences and word learning, followed by discussion of potential implications of this research for embodied theories of action and language integration. Index Terms— visual experiences, word learning, parentinfant interaction, embodied cognition. I.

INTRODUCTION

Cognitive robotics has recognized the importance of close interactions between language and action and recent models proposed to ground robot language understanding in sensory-motor representations. These models, however, focus mainly on the one-way relation between language and action – that is, how language can be grounded in sensorymotor representations. The complex loops among sensorymotor systems, visual object recognition and word learning have not been explored yet. Hence, the primary aim of this tutorial is to present recent developmental psychology studies, while advocating an integrated approach to world learning. This special session will present recent empirical evidence on developmental dependencies between motor developments, action on objects, visual object recognition and object name learning in young children, followed by considerations of the potential implications of this approach for robotics.

978-1-4799-7540-2/14/$31.00 ©2014 IEEE

Anna M. Borghi University of Bologna Bologna, Italy [email protected]

II. INFLUENCE OF LANGUAGE DEVELOPMENT ON MOTOR BEHAVIOR

Beata J. Grzyb, Allegra Cattani, Angelo Cangelosi, and Caroline Floccia Young children sometimes fail to use information about object size and make a serious attempt to perform impossible actions on miniature objects. Examples include children trying to sit in a tiny chair, put doll shoes on their own feet, or get inside small cars. We investigate the nature of the scale errors and their potential relation to language development. The possibility we explore here is whether the relative proportion of descriptors (nouns and adjectives) predicts the extent to which a child is likely to produce a scale error in a lab situation. We will also present another important aspects of the current Babylab research. The first one investigates the structure of the lexical network in young toddlers either by using a combination of visual and auditory stimuli, or by focussing on purely auditory stimuli. We know that in adults, the words in our lexicon are tightly interconnected, semantically (sock/jumper), associatively (cat/milk), phonologically (cat/bat), etc. How does this lexicon start? Do children learn these words in isolation from one another? Or as a miniature network? And if, so, what is the nature of the first links? The second work investigates the interlink between first words learning and gestures across children from different cultures with the use a controlled naming task. Gestures are often seen as reliant on certain cultures but they can also support children’s language development. We will demonstrate that gestures act as a prime for subsequent word learning.

6

III. SOCIALLY GENERATED VISUAL EXPERIENCES AND WORD LEARNING

Hanako Yoshida, Joseph Burling Mothers of young infants actively handle objects for showing. They may generate general actions and gestures with objects while labeling them, and also incorporate other specific sounds and motions related to the object during infant play. Also, the onset of referential communication and word meaning comprehension has been recently documented as early as at 6 months. However, the dynamic processes that link visual experiences with social input and their role for language learning has not been systematically addressed. As a first step, the present study will look at the dynamic relationship between acting on objects, viewing these objects, and the role of object labeling during parent-infant play sessions. Observations were made longitudinally, with assessments at 3—18 months, every 3 months. These experiences were captured by a head mounted eye tracking device that recorded scene interactions from the infant’s momentary eye gaze. We ask: 1) does parental object naming change over time, 2) is there any relation between parent handling objects, size of the object, and parent labeling, and 3) potential relevancy toward the development of vocabulary knowledge. Our preliminary evidence indicates that parents name objects regularly across early infant development, and these naming moments were associated with object characteristics such as size. Also, some indication of the relevancy of vocabulary scores to object manipulation was observed. We will discuss the significance of the findings in the framework of the embodied nature of early language learning experiences. IV. ACTION AND LANGUAGE: A FLEXIBLE AND DYNAMIC INTERACTION

Anna M. Borghi I will overview some recent work in our lab highlighting the bidirectional and flexible influence between language and action. First, I will illustrate some studies showing that, during language comprehension, affordances are activated in a flexible and context dependent way. In addition, I will briefly describe studies showing that producing words to ask for objects influences the bodily perception as it leads to an extension of the near space, as it happens with tools. Implications of these studies for embodied theories of action and language integration will be discussed. V. EARLY VISUAL FUNDATIONS FOR ACQUIRING OBJECT KNOWLEDGE

Swapnaa Jayaraman and Linda B. Smith Infants show an understanding of common nouns – looking to the named thing – by as early as 9 months. Although there has been much study of the statistical structure of words infants and young children hear that 978-1-4799-7540-2/14/$31.00 ©2014 IEEE

support this learning, we know very little about the corresponding visual experiences. In this presentation we consider how we may collect and study the statistical structure of early visual experiences. What are the opportunities that infants get to experience these visual objects? How can we characterize these opportunities? Is there a structure in these early environments, do they change with age, and, if so, how? The workshop will present examples from our project to use head cameras to collect the natural visual environments of infants. These videos provide a measure of the availability of visual objects in the typical environments of infants and allow us to characterize their early experiences. Findings (principally from infants 1 month to 12 months of age) reveal that the visual environments of infants are highly structured and ordered in time. Early environments are rich in face experiences with signature properties that seem ideal for learning. Later environments slowly start offering excellent views of hands – that point to, touch, and grasp objects – that might be instrumental in object learning and recognition. We will also discuss some lessons learned – methodological and analytic –from this effort to collect and study a corpus of first-person scenes from infants. VI. VOCAL AND MOTOR COORDINATION IN JOINTCONSTRUCTION TASKS DURING PARENT-CHILD SOCIAL INTERACTIONS Alfredo F. Pereira, Isabel C. Lisboa, Emanuel Sousa, Jorge A. Santos, Wolfram Erlhagen, and Estela Bicho Social coordination can be measured at a time scale of less than a second by the partners’ synchronization of sensorimotor activity and at a larger time scale (of a few seconds to minutes) by the oscillation of activity (turntaking) and the modulation of the interaction rhythm, evident in the statistical contingencies between the two. Recent studies of children’s real-time sensorimotor experience, during social interaction, suggests that motor activity – in particular head and hand movements – organizes the interaction, scaffold learning, and index the quality of turn-taking. We will present the findings of a new study that examined vocal and motor coordination between parents and their children (aged 3-5 years), while they were engaged in two joint construction tasks: (1) based on a model in a photo, assemble an object from blocks; and (2) build a tower with blocks. Participants’ head and arms were motion-captured and the interaction was videotaped. For measuring vocal coordination, we followed the approach in studies of mother-infant’s attachment bond development and extended it to motor behavior. Results suggest that vocal and motor coordination are modulated by task complexity – measures of coordination show a more tightly coordinated interaction for the most complex task, the copy task (1).

7