A Wizard-of-Oz environment to study Referring Expression Generation in a Situated Spoken Dialogue Task Srinivasan Janarthanam & Oliver Lemon University of Edinburgh www.classic-project.org

Data

Introduction We present a wizard-of-oz environment to collect dialogue corpus for adaptive Referring Expression Generation (REG).

User User utterance (audio)

Wizard Interface Tool

System utterance (audio)

• To study how users with different domain expertise react to different referring expressions. • To study the effect of lexical alignment of the system to users due to priming. (Pickering and Garrod, 2004) • To build statistical user simulations sensitive to REs to learn adaptive natural language generation (NLG) policies. (Janarthanam and

Wizard

User DA

Dialogue System

NLG module System DA

(Jargon/Desc/Tutor)

System utterance

Speech Synthesis (Cereproc)

Lemon, 2009) DM Strategy

Task To setup an Internet connection by connecting various components (broadband filters, modem and the computer) using broadband and ethernet cables. The system will give step-by-step instructions to setup the connections.

User environment

Fig.1. Wizard-of-Oz environment

Strategies

Wizard-of-Oz environment

•Jargon – Use technical terms as referring expressions

• Wizard Interaction Tool - System Response Panel - Confirmation Request Panel - Confirmation Panel - User utterance Panel - User RE choice Panel

•Descriptive – Use descriptive referring expressions

“Connect one end of the broadband cable to the broadband filter.” “Connect one end of the thin cable with grey ends to the small white box.”

•Tutor – Use both to teach technical terms “Connect one end of the broadband cable to the broadband filter. The broadband cable is the thin cable with grey ends. The broadband filter is the small white box.”

• Instructional Dialogue Manager - Instructs the user by giving step-by-step instructions - Follows a deterministic dialogue management policy - Handles clarification requests

Preliminary analysis • 17 participants (Jargon – 6, Descriptive – 6, Tutorial – 5) • All strategies had similar task completion rates. • Utterance Length – Jargon utterances are shorter than descriptive and tutorial utterances. Tutorial utterances are the longest. • Learning Gain - Tutorial dialogues had the most learning gain followed by Jargon. Descriptive dialogues had the least learning gain. • Dialogue Time – Descriptive dialogues had the shortest dialogue time followed by Jargon dialogues. Tutorial dialogues are longer. • Dialogue Turns - Tutorial and Descriptive dialogues had the shortest number of turns (due to fewer clarifications). Jargon dialogues are often longer because of clarification requests. • User response time - Expert users had faster response times than Novice users in Jargon dialogues.

Can adaptive referring expression generation policy learn to combine the benefits of the different strategies ?

• User - Interacts with the dialogue system using a headset. - Follows the instructions given by the system. - Not aware that his utterances are interpreted by a wizard. • Wizard activities - Listen, interpret and annotate the user’s utterance - Request/provide confirmation - Record user’s novel RE usage

• User background survey • Knowledge pre-test – User’s initial domain knowledge survey • WIT logs – Logs system and user dialogue acts, utterance time, referring expression choice. • User utterances audio file. • Knowledge gain post-test – User’s domain knowledge after the task. • User satisfaction survey – Questionnaire on various system features. E.g. “Conversation was easy”, “It was easy to identify the domain objects”, etc. • Task completion – User’s internet setup is examined for task completion.

References

Fig.2. Wizard-of-Oz Interaction Tool

S. Janarthanam and O. Lemon. 2009. Learning Lexical Alignment Policies for Generating Referring Expressions for Spoken Dialogue Systems. In Proc. ENLG’09. M. J. Pickering and S. Garrod. 2004. Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences, 27.

12th European Workshop on Natural Language Generation March 2009, EACL 2009 Athens © Srinivasan Janarthanam & Oliver Lemon