Automated Product and Service Domain Question Answering Ajeet Parhar, Jason Jiang, Greg Findlow, and Ming Zhao Telstra Research Laboratories 770 Blackburn Road, Clayton Victoria, 3168, Australia {Ajeet.Parhar, Jason.Jiang, Greg.Findlow, Ming.Zhao}@team.telstra.com Abstract— This paper presents the features of Laurel, a

restricted domain Question Answering system being engineered to answer Product and Service natural language questions. Development of Laurel follows an iterative approach which i s presented here. The current iteration is the subject of this paper. It employs light weight natural language techniques and introduces Answer Extraction Trees to co-ordinate answer identification. We also present our analyses of approximately 1000 Product and Service questions underlying the current iteration’s gold standard test data.

Keywords— Question Answering, Products and Services INTRODUCTION roducts and services (P&S) are fundamental to the commercial world and their very existence creates a need to access information about them. From a customer’s perspective this information may support purchasing decisions, through to advising on normal operation and in cases of fault. Automating product and service information acquisition with real time on-line question answering (QA) techniques offers advantages over keyword and search engine methods in its potential to provide focused responses to focused queries. Question answering literature has emerged in a number of contexts, such as in traditional AI (eg. [Lehnert, 1978]), in TREC open domain question answering challenges (eg. [Voorhees & Buckland, 2005]) and in various restricted domains (eg. [Molla & Vicedo, 2004]). The tools and techniques reported there, while seen to be partly problem dependent and partly problem independent, rarely specifically addressed the P&S area which is characterized by the forms and types of questions asked and the nature of language used in the domain documents1. This paper presents the first iteration results of our ongoing P&S QA system engineering activities. We characterize the P&S QA problem as follows. Let p be a domain (of one or more products and/or services), Dp be the set of documents associated with p and Qp be the set of I.

P

questions asked of an ideal QA system, _, related to p over the lifetime of p. Our aim is to develop _ to successfully operate over a range of domains, p1 … pn. The questions that relate to a domain are central to the construction and testing of a QA system. If Q p and D p are known, then _ can in principle be constructed2. However, Qp cannot be determined until _ is constructed. To resolve this dilemma we take the following approach: (1) estimate the ideal set of questions, Q p, then (2) use Q p to construct an approximation of an ideal QA system, _´, that answers all the questions in Q p then (3) use _´ to obtain a better estimate of Qp to (4) construct a better _´ and so on until (5) some stopping criteria is met. This process assumes the _´ converge to _, which we find uncontroversial. However, in practice the _´ developed in (2) and (4) is unlikely to accurately answer all of the questions in their immediately prior steps (and likely to answer a wider range of questions)3. We believe that if (1)-(5) is followed with the intent of developing Qp and particularly _, the effect of the inaccuracy will only be to slow the rate of convergence. In the following section we present the results of collecting and analyzing question for two P&S domains (step 1 above), and discuss these results in relation to other restricted domain findings. Section 3 presents the results of designing, developing and testing a web based QA system engineered to address the collected questions (step 2 above). Following this we discuss overall findings and present our future directions. II. QUESTION ACQUISITION A. Harvesting Questions To build an evaluation data set we selected two test domains, referred to hereafter as MOBILE and CITYLINK . The MOBILE domain relates to mobile products and services offered by a large telecommunication carrier, while CITYLINK consists of customer service information for motorists using an electronic tolling system employed on a number of the freeways in an Australian city.

1

Prior investigation into the medium of accessing a QA system highlighted that web, e-mail and voice access for P&S QA purposes significantly affected the form and type of questions asked. Also the degree to which the user expects dialogue to be handled affects the level to which questions are self contained (we assume no dialogue handling in the work reported here).

2 E.g. by the uninteresting approach of manually answering the questions in Qp and imbedding these into a QA system, _´. 3 Testing a non-perfect _´ against an estimate of Qp gives a measure of linguistic performance, while testing against Q p and question frequency information gives a measure of user experience.

The source documents for the MOBILE domain comprised about 5000 words. They were written in an informal marketing style. The documents were divided into 162 passages (including text paragraphs as well as headings, subheadings, etc.) for the purposes of processing. The source documents for the CITYLINK domain comprised about 3500 words and were written in a more formal style. These documents were divided into 70 passages. Importantly, both sets of documents contained little redundant information. To estimate Qp for these domains we chose 20 people randomly from a large group of candidates without formal linguistics training. They were asked to read each set of domain texts. At the end of each main section they were asked to write a few questions answerable by the text in that section, while bearing in mind the intended use of their questions. Overall they were asked to author at least 25 questions for each of the two domains. This activity produced 1200 questions. Some of these were removed from the final data sets due to either no answer being in the domain documents or language use being too different from standard Australian English4. The final sample sets comprised 509 questions for the MOBILE domain and 522 questions for the CITYLINK domain. B. Question Analysis The collected questions were manually inspected to understand their nature, variations and distribution, and in turn identify classes that may aid the QA process. Seven primary classes were delineated and further subdivided into 20 subclasses as shown in Table 1. Here bold entries refer to classes, normal font to subclasses and example questions are in italics. The percentage numbers refer to the proportion of questions in that class/subclass. Our choice of question classes/subclasses differs from a purely question stem oriented division [Pasca 2003] that is prevalent in QA literature. This is due to our motivation to identify question types which occurred frequently in the test domains (e.g. distinguishing “Special Type factoid” from “Factoid - other than special type” reflects observed frequency), and the desire to minimize the complexity of the Question Processing module which operates in real-time. That is, the classification is not solely syntactically based. An observation of note is that factoid answer questions comprise less than half of the sample data, highlighting the limitation of only using TREC published techniques to address the overall problem. Another observation of the question set was that the questions had relatively simple syntactic structures (there were no complex questions [Harabagiu & Lacatsu 2004] with only 5% not beginning with a question stem). Consequently, a small number of regular expressions could be used to reliably classify the questions. We also observed that yes/no questions were prevalent and had not been significantly addressed previously in QA literature. An observation not apparent from Table 1 was that there wasn’t a strict adherence to the use of product names in questions (nor in the domain texts) implying that QA techniques focusing on terminology in terminology rich domains (e.g. [Rinaldi et al. 2004]) may not suit this situation. For comparison, P&S QA work reported by Heigh & Kosseim [Molla & Vicedo 2004] used 140 questions and 4

Both issues were deferred for future work

220 documents (560K characters) comprising one domain covering multiple services. Table 1 Showing the question classification scheme and distribution of questions across the two domains. Factoid (other than special type) (29%) Location/position factoid questions (6%) Where can I use my mobile phone to buy a coke? Person/org. Qs with reference to phone contact (1%) Who do I call for more information on WAP? Other person/organization faction questions (1%) Who is eligible to use mobile phone parking? Time point/interval factoid (including “When…”) (5%) At what times are the citylink passes valid? Generic quantity factoid questions (2%) How many e-tags can I get in one account? Factoid/degree “how” Qs (2%) How big is an etag? Other NP-embodied factoids (NP defines ans. type) (12%) Which banks can I use with Mobile EFTPOS? Special type (factoid) (10%) Cost Qs (8%) What charges are incurred using WAP? Contact phone number Qs (2%) What is the number of the customer hotline? ‘Meaning’ questions (1%) Questions asking what something ‘means’ (1%) What does it mean when my e-Tag beeps? Definition or definition-related (1%) Explicit acronym definition Qs (<1%) What does WAP stand for? Difference between 2 or more terms etc. (1%) What is the difference between activation and, registration? “What” Qs not previously covered (13%) “what” with a probable non-factoid (VP) answer (<1%) What has CityLink done to reduce traffic noise? “what” with a probable factoid (NP) answer (3%) What do I need to make mobile fax & data calls? “what is/are” questions (10%) What's the speed limit in the tunnels? Yes/no (29%) Existence/availability yes/no questions (1%) Is there a connection/flag fall fee for WAP? Other (general) yes/no Qs (28%) Are refunds available if I don't use my CityLink pass? Longer answer (15%) Why Qs (2%) Why do I need to register to access WAP? How Qs (13%) How do I use m-Commerce parking? What Qs requiring longer answers (5%) What happens if I have reached the pass limit? C. Qp and Evaluation Data Set Having collected questions from our test subjects, we then created an evaluation data set suitable for testing the overall performance of a QA system. This involved manually identifying answers to those questions in the source documents. For each question a single sentence was sought as

the best (or at least equal best) answer to that question. For the majority of questions, this could be unambiguously identified, however for a number of questions we had to select a sentence or text segment which, rather than providing the actual answer, constituted a best entry point in the document from which the user could begin reading to find the answer. For example, for some mobile phone services, costs were specified across two or three sentences, depending on which network the user was on. In such a case we selected the heading of the smallest (sub)section containing the answering sentences to be the “single best answer sentence”. III. SYSTEM DESCRIPTION A. Overview Following the creation of a reference question and answer data set a first iteration question answering system was developed. Its design is a variant of the pipeline structure frequently seen in TREC focused question answering systems [Harabagiu & Lacatusu 2004] (see Figure 1). It comprises two streams of processing, real-time and off-line. Raw documents are input to the off-line Document Processing component to add indexing and other information to enable later query matching. Real-time processing comprises three main activities: Question Processing, Passage Retrieval and Answer Extraction and Formulation. When given a question, the question processing function aims to extract information that will aid subsequent processing. This information is used in the passage retrieval function to identify a limited set of passages that are likely to contain the answer, and in the answer extraction function (which aims to identify the smallest possible segment of answering text) to help identify the correct answers from within those passages. Question

Documents

Question Processing

Document Processing

Passage Retrieval

Document Store

Answer Extraction And Formulation Answer

Figure 1 A context diagram of the QA system. We view the distribution of computational effort between the Passage Retrieval and Answer Extraction modules, and between on and off-line processing as variables in an overall QA system design. For example, in TREC oriented systems large numbers of documents (~106) result in light document processing and a clear separation between Passage Retrieval and Answer Extraction, while the lesser number of documents in domain specific applications (~103) allow more sophisticated document processing and less distinction between Passage Retrieval and Answer Extraction.

B. Question Processing The Question Processing module “extracts” information from the question text to populate the dimensions of a question feature vector. For example, POS tags constitute a dimension, as does chunk parser output [Jiang & Rowles 1995] and question classification according to the categories in Table 1. Dimensions depend on each other as well as the question text. For example, the classic answertype dimension only exists and is populated for factoid answer classified questions. C. Document Processing and Passage Retrieval The Passage Retrieval module selects, from a document collection, a short list of text passages that are likely to contain the answer to the question. In restricted domains such as the two test domains, the answer to a given question often occurs in only one passage (typically a paragraph). Consequently it is important for the Passage Retrieval module to employ techniques to maximize its recall. For a long time, there has been a great deal of interest in applying natural language processing (NLP) techniques to information retrieval (IR) tasks. However, in general, the results have not been encouraging, typically due to lack of robustness and inefficiency, particularly when working with large numbers of documents [Jones 1999]. Since we expect to deal with relatively small document collections offline and high recall is very important for the Passage Retriever, we decided to test robust light-weight NLP index term generation techniques to see if there were retrieval gains over using a traditional statistical IR method. Indexing terms for a passage5 are generated as follows: i. Apply a regular-expression-based preprocessor to replace specific lexico-semantic terms/expressions (e.g. product names and monetary values) with labels; ii. Derive a set of open class words and phrases (NPs, VPs, adjectives and adverbs) by tagging each preprocessed sentence using a Brill Tagger [Brill 1995] and passing them to a rule-based chunk parser [Jiang & Rowles 1995]. Insignificant words are removed from the phrases (e.g. “the”, “your”); iii. Create a paragraph index by listing the words from ii., their stemmed counterparts (using a Porter stemmer) and terms/expressions identified in i; iv. Calculate the term frequency (TF) of an index term w.r.t. the paragraph, then weight it using the following [Manning & Schütze 1999]: f(TF) = 1 + log(TF) (if TF > 0, otherwise 0) v. Create a table in which index terms are associated with all the paragraphs IDs containing that term together with the corresponding f(TF) of the term. For each word in the document collection, the system also assigns an importance score which reflects the overall significance of the word in the collection. This is based on the average similarity between paragraphs with and without the word and is determined by summing the similarities (using a cosine coefficient [Salton 1971]) between each paragraph and the average of the word frequency vectors of all the paragraphs in the collection. 5 P&S documents often contain headings and subheadings that provide important topical information about the paragraphs under them. To capture this, document processing initially adds each heading to the beginning of each of the passage chunks (typically paragraphs) under it.

Given a question, the indexed paragraphs and importance scores, the passage retrieval process occurs as follows: 1. Apply the passage index term generation process (i-v) to the question to obtain a set of search terms. 2. For each search term, identify all the index terms sharing at least one word with the search term. For each selected index term, calculate a . The ratio of the number of search words to the number of words it contains, which indicates how closely the index term matches the search term; b. The sum of the importance scores of the words shared by the search term and the index term. 3. Multiply A and B to get a figure that we call the fitness value of the index term wrt. the search term. 4. Multiply the fitness value of an index term and its weighted TF in each passage to get the matching score of the index term for the passage wrt. the question. 5. Compute the sum of the matching scores of all the index terms for each passage. Sort the passages according to their total index term matching scores. Return the first N passages as the results of passage retrieval. We compared the performance of this passage retrieval method with a baseline statistical information retrieval method [Jiang et al. 2002], modified to not penalize the system if there is more than one correct answer in the document collection [Burke et al. 1997]. Table 2 and Table 3 shows the results for the CITYLINK domain (70 passages, 522 questions) and MOBILE domain (162 passages, 598 questions) respectively. The first rows show the number of passages returned. The second and the third rows show the percentage of questions that have at least one answer-providing passage in the returned passages. The retrieval method with lightweight NLP based index terms performs consistently better than the baseline for the CITYLINK domain. The results for the MOBILE domain were not as differentiated. Overall, the lightweight NLP methods are seen as increasing performance. Table 2 CITYLINK domain passage retrieval results. Num. Psgs 1 3 5 7 10 20 Baseline

(%) WithNLP (%)

59.5

69.8

75.6

80.4

87.8

91.1

36.5

60.2

69.4

77.5

83.0

91.7

96.7

Table 3 MOBILE passage retrieval results. Num. Psgs 1 3 5 7 10 Baseline

(%) WithNLP (%)

30

33.5

20

30

23.0

42.1

53.0

61.2

68.4

78.3

81.7

24.2

41.5

55.2

60.8

70.2

79.5

83.1

The extra computational cost for using the lightweight NLP method was reflected in the retrieval time being 20-30 times longer than when using the statistical approach. Even so, it was well within the “real-time” range. Following these experiments an improved method of exploiting document structure information to enhance the passage retrieval was developed (reported in [Findlow 2005]). Future directions include use of an n-gram based feature extractor [Jiang et al. 2002] to calculate the weight of an index term in a passage to replace its weighted TF.

D. Answer Extraction 1) Answer Extraction Trees The Answer Extraction and Formulation module is given a question feature vector and a list of passage feature vectors and aims to output a prioritized list of candidate text answers. To perform this task we introduce the notion of an Answer Extraction Tree (AET). This follows the observation that many techniques can be employed to match question features to passage features for answer identification. Each technique performs best with a particular combination of question and passage types. For example, “What number do I call to report a fault?” relates to a short succinct answer (factoid), a phone number, instances of which can be located in passage texts and analyzed with surrounding text to determine answer candidacy. On the other hand a “how-auxiliary” question, such as “How can I contact the help desk?” relates to the manner of achieving a goal, which generally involves a non-factoid answer and comparatively more complex answer text identification. Instead of developing a single technique applicable to all situations we felt it would be better to engineer a desired level of performance for particular categories of question and passage text features, through the use of a coordinating framework. An AET is an abstract computational structure that comprises directed arcs, fork nodes, leaf nodes, and default nodes. A root node is a distinguished fork node that has no arcs leading to it. Fork nodes are linked to other fork nodes and leaf nodes with directed arcs. Fork nodes are also associated with a default node. Processing is associated with leaf nodes, default nodes and arcs connected to fork nodes. Arcs originating at fork nodes are associated with functions of the form A: Q _ P -> {True, False}, where an element of the set Q is a vector of question features and an element of the set P is a list (including the empty list) of vectors of passage features (one vector per passage). A True response indicates that the arc can be traversed, while False indicates that it cannot. Leaf and default nodes perform functional evaluations of the form N: Q _ P -> {list of passage subtext vectors}, where Q, P are as above and output is a list of passage subtext vectors (text from passages, passageId, etc.) that represent candidate answers to the question in preferential order. The root of our AET is depicted in Figure 2. The arc labels indicate the name of their evaluation function, while the percentages indicate the proportion of evaluation set questions (in both domains) proceeding along that arc. In this figure arc functions only consider question feature vectors (ie. passage information is unused). For example, the yesNoAnswer function tests for the existence of a ‘yes/no’ answer question. Processing of an AET begins at the root node. The arc functions leading from this node are evaluated in sequence until either a True response is obtained, and the arc is traversed, or if all the arc functions return False then the default node is reached. If an arc is traversed then the above activity is repeated until a leaf or default node is reached. Answer extraction processing typically occurs at the leaf nodes. If at any stage in the traversal of the AET there is not enough evidence to support the use of a more specific technique (i.e. an arc traversal from that node) then a default technique associated with the default node is invoked.

AET Root characterisationAnswer 14% 28% 39% factoidAnswer yesNoAnswer

default

1%

how-auxAnswer 5% 13% longAnswer whyAnswer

Figure 2 Showing a representation of the root level of the Answer Extraction Tree The advantage of this scheme is that system performance can be engineered by extending the AET structure to deal with increasingly specific situations as well as readily restructured to cater for new broadly applicable question answering techniques. The subsequent levels of the AET developed in this project and details of leaf node functions are not further discussed here. 2) Wide Coverage Default Processing We found it convenient to use a generic default processing function (WDP – wide coverage default processing function) for all nodes of the AET prior to developing more tailored default processing functions. QA performance based on the WDP alone represents a worst case scenario as the addition of more specific default and leaf node functions by design increases system performance. Our WDP aims to identify co-ordinations of key question concepts6 in passage text as candidate answers. We justify this as follows. A strong candidate answer for a factoid answer question is formed by transforming it (e.g. wh-movement and aux-inversion transformational grammar transformations) to a statement, q-statement, and substituting the answer type expression for its correct instance. E.g. the following qstatement, “The [train]noun chunk head [leaves]verb chunk head for [Alice Springs]noun chunk head at [noon]answer type instance.” would be a very strong answer candidate for the question “What [time]AT does [the train]N [leave]V for [Alice Springs]N?”. The q-statement is, of course, unlikely to occur in a passage, so instead one can look for text that is increasingly remotely similar to the q-statement to form a principled technique for default answer text identification7. E.g. the following would answer the question above even though the key concepts are not all in the one sentence, and the terms are not all identical to those of the q-statement8: “The [Ghan]noun chunk head [departs]verb chunk head at [noon]answer type. It will stop at Tennant Creek and [Alice]noun chunk head”. In these sentences the bracketed text are the key elements of the co-ordinations. While this justification is based on factoid answer questions, we found that it produced acceptable default results for all question types in our evaluation data. This appears to follow from the nature of questions asked, lack of redundancy in answering passages and paucity of negation and quantification terms in the domain documents. Our WDP identifies co-ordinations by passing each passage sentence through a decision tree that asks the 6 Identified as question terms and relevant synonyms, hypernyms, etc. of question NP & VP heads, and additionally for factoid questions, answertypes. NP and VP heads are readily identified from chunk parser output. 7 This assumes that the concepts reflecting the q-statement’s verb and noun chunk heads are likely to be nearby in text containing the same meaning as the q-statement. 8 ‘Ghan’ is a category of train; ‘depart’ is a synonym of ‘leave’ and ‘Alice’ is a contraction of ‘Alice Springs’.

following series of questions: (1) Does the passage contain the answer type? (2) If so, is it in this sentence? (3) If so, does the passage contain at least one of the question noun head chunks? (4) If so, is it in this sentence? (5) If so, does the passage contain at least one of the question verb head chunks” (6) If so, is it in this sentence? A “no” answer at any point leads to discrimination in a correspondingly reduced subtree. Each leaf node of the decision tree effectively represents something about the distribution of noun and verb chunk heads (and if present, answertypes) in the passage, relative to the sentence. The sentence being evaluated is then given a score according to a leaf node ranking scheme which maps each decision tree leaf node to an integer. The leaf node corresponding to the most likely candidate path (answer type instance present in the sentence, and at least one noun chunk head and one verb chunk head are present in the same sentence) was given the highest score, while the leaf node corresponding to the least likely candidate path (no answer type instance, noun chunk head or verb chunk head is present in a passage) was given the lowest score regardless of ranking scheme. The highest sentence score in a passage becomes the passage score. The passages of equal rank are then sub ranked according to numbers of noun chunk head matches, then verb chunk head matches9. Table 4 shows WDP passage ranking results over all evaluation set questions for the cases where the answer extraction and formulation function was given a list of passages that included the answer containing passage10. The answer containing passage was in the first five ranked passages about 75% of the time with the answer containing passage ranked first 39.6% of the time. Table 4 WDP passage rankings over all evaluation questions Passage Rank 1 2 3 4 5 Occurrence (%)

39.6

14.5

10.4

4.6

4.6

The WDP implementation’s appeal is in its extreme simplicity and speed. Its current limitations include ignoring the actual conceptual relationships between NP heads and VP heads, and ignoring adjectives and adverbs as well as the influence of negation and quantification. Addressing such issues requires the availability of additional information and analysis (e.g. case and role [Gildea & Jurafski 2002, Macleod & Grishman 1994]). Also the use of pseudo-logical transformation techniques [Harabagiu et al. 2000, Mollar & Gardiner 2004] would enable more specific queries to be associated with decision tree nodes.

9 Noun chunk head matches were observed to be far more significant answer indicators than verb chunk head matches. 10 Up to 35 passages were being ranked.

E. User Interface Users access Laurel via a web browser. Questions are typed in a window and responses are displayed below it. The displayed answer is the smallest passage text, at the granularity of sentences, computed to answer the question. The user has the option of viewing the context in which the answer is stated [Lin et al 2003] by seeing “more” of the text surrounding the initially displayed sentence(s) via a button press. If one sentence is initially displayed, “more” refers to the sentences, which contain the key question concepts, before and after it. If a few sentences are initially displayed, then “more” refers to the entire passage. Subsequent iterations of gold standard question acquisition are expected to involve this user interface. IV. DISCUSSION In the P&S domain documents addressed here we observed that questions were typically likely to be answered in a single place in a single document. This contrasts with a number of TREC examples we have seen quoted in the literature, where there are many correct answering text segments present. An effect of this redundancy is to increase the likelihood of finding an answer using identical terms to the question. For example, “managing director of Apricot Computers” might be found in the process of responding to the question “Who is the managing director of Apricot Computers?”. Consequently it is relatively more important for P&S QA to address the mapping of user question terminology to that used in domain documents. We also observed that underspecification of questions was an issue of note. For example, the presence of the deictic expressions “I” and “my phone” in the question “Why can’t I use my phone to pay for parking?” ideally required the QA system to have or gain knowledge of the user’s situation to provide a naturally limited answer scope (e.g. do they have a pre- or post-paid mobile phone, which network are they using, etc.). As 60% of the evaluation set questions contained either “I” or “my”, and in many cases assuming a typical or average customer was inappropriate, there is a strong case to incorporate clarification dialogue handling in future iterations. V. CLOSING REMARKS In this paper we presented the iterative method used to develop the Laurel question answering system, as well as the main details of Laurel’s design and operation. The role of a gold standard question and answer set was central to the development. Iterative improvement of the gold standard and question answering system together are seen as underlying a direction forward in developing more capable restricted domain question answering systems. ACKNOWLEDGEMENTS We would like to thank Peter Sember for his comments on earlier versions of this paper. The permission of the Chief Technology Officer of Telstra Corporation Limited to publish this paper is hereby acknowledged. REFERENCES Brill, E. (1995) Transformation-based error-driven learning and natural language processing. Computational Linguistics, 21(4):543—565

Burke, R. D., Hammond, K. J., Kulyukin, V., Lytinen, L. L., Tomuro, N. & Schoenberg, S. (1997) , Question Answering from Frequently Asked Question Files: Experiences with the FAQ FINDER System – Internet. AI Magazine, Summer. Findlow, G. (2005) Document Structure Based Passage Retrieval for Question Answering. Paper submitted to PACLING2005, Tokyo. Gildea, D. & Jurafski, D. (2002) Automatic Labeling of Semantic Roles. Computational Linguistics, 28(3), pp:245-248 Harabagiu, S. & Lacatusu, F. (2004) Strategies for Advanced Question Answering, In Proc. Pragmatics of Question Answering Workshop, Boston, MA, May 6-7. Harabagiu, S. Maiorano, S. & Pasca, M. (2000), Experiments with OpenDomain Textual Question Answering. In Proceedings of the 18th International Conference on Computational Linguistics (COLING2000), Saarbruken Germany, Aug., pp:292-298 Jiang, J. & Rowles, C. (1995) Robust Natural Language Query Processing Using Key-Centered Phrase Structure Frames. Proceedings of Eighth Australian Joint Conference on Artificial Intelligence. Canberra. Jiang, J., Starkie, B., & Raskutti, B. (2002) , An Information Retrieval System. WO0233583A1, International patent, World Intellectual Property Organisation (WIPO). Jones, K. S. (1999), What is the Role of NLP in Text Retrieval?, In Strzalkowski, T. (eds) Natural Language Information Retrieval. Kluwer Academic Publishers, The Netherlands, pp:1-24. Lehnert, W. G. (1978), The processing of question answering. Lawrence Erlbaum Associates, Hillsdale NJ. Lin, J., Quan, D., Sinha, V., Bakshi, K., Huynh,D., Katz, B. & Karger, D., (2003) The Role of Context in Question Answering Systems, Proc. Tenth Text Retrieval Conference (TREC 2001). Macleod, C. & Grishman, R. (1994) COMLEX Syntax Reference Manual Version 1.2. Linguistic Data Consortium, University of Pennsylvania. Manning, C. D. & Schutze, H. (2001) Foundations of Statistical Natural Language Processing. MIT Press, London. pp:542. Molla, D. & Vicedo, J. L. (eds.) (2004) Proc. Question Answering In Restricted Domains ACL Workshop, Barcelona, Spain, 25th July. Molla, D. & Gardiner, M. (2004), Answerfinder: Question Answering by Combining Lexical, Semantic and Syntactic Information, Proc. Australalsian Langauge Technology Workshop (ALTA 2004), Sydney, Australia. Rinaldi, F., Hess, M., Dowdall, J., Molla, D. & Schwitter, R. (2004) Question Answering in Terminology-Rich Technical Domains. In Maybury (ed) New Directions in Question Answering, AAAI Press/The MIT Press, Menlo Park, CA. Salton, G. (1971) , The SMART Retrieval System – Experiments in Automatic Document Processing, Prentice-Hall, New Jersey. Simmons, R. F. (1970), Natural Language Question-Answering Systems: 1969, Comm. ACM 13, 15 Voorhees, E.M. & Buckland, L. P. (2004) The Thirteenth Text REtrieval Conference Proceedings (TREC 2004), National Institute of Standards & Technology.