Statistical Machine Translation

Term Project 1. Don’ts-to-Do’s English-English Decoder 2. Chinese Spell Check Decoder 3. Natural User Interface to Linggle 4. Term Translator Jason S. Chang National Tsing Hua University 2013-0527 1

Entire contents © 2012 Jason S. Chang. All rights reserved.

13年5月27日星期一

3. Natural User Interface to Linggle

2

Entire contents © 2011 Jason S. Chang. All rights reserved.

13年5月27日星期一

Using MT decoder to handle NUI input • Scenario: convert NUI input to command • What are some words that describe a fork? --> $N fork --> $A fork • Training – language model • Compute pos-word ngram based on Web 1T 5-grams – $N spoon 9999, $A spoon 9999 – translation model • Crawl answer.com • Obtain QA pairs – Q = What are some words that describe a spoon?' – A = silver silver plate silverware stainless plastic wooden soup dessert utensil... – dominant pos in A = noun • add (what are some words that describe a ||| $N ||| 0.0) to data/tm 3

Entire contents © 2011 Jason S. Chang. All rights reserved.

13年5月27日星期一

Data and code related to decoder • Data – data/input ── test sentence (NUI sentences) – data/lm ── language model (commands ngrams) – data/tm ── mapping between NUI sentences and commands • Code – decode.py ── INPUT bad English sentence (data/input) ── OUTPUT bad/good English sentence (data/output) – models.py ── INPUT language+translation models (data/lm+tm) ── OUTPUT dictionary of LM and TM

4

Entire contents © 2011 Jason S. Chang. All rights reserved.

13年5月27日星期一

Training data

5

• Data – bncWordLemmaPos.txt ── BNC with lexical information – 2ga.txt, 3ga.txt ──Web 1T ngram filtered (GSL+AWL) • Training – language model • use the Web 1T to generate commands • e.g., sandy beach --> Linggle commands – $A beach – sandy $N • generate language model from Linggle commands – translation model • use qestion-answer pair • map question phrase to dominating part of speech in answer • generate (question phrase, command symble, probability)

Entire contents © 2011 Jason S. Chang. All rights reserved.

13年5月27日星期一

Finding Dominating POS • Data – bncWordLemmaPos.txt ── BNC with lexical information • Code – findAnswerPOS.py ── 計算 answer 部分的主要詞性 • A = Some words that can describe the noun spoon are: silver silver plate silverware stainless plastic wooden soup dessert utensil... • OUTPUT = noun

6

Entire contents © 2011 Jason S. Chang. All rights reserved.

13年5月27日星期一

Using MT decoder to handle NUI input • Modify decoder – modify lm • add $N spoon, $A spoon, $N fork, $A fork in 2-gram • add $N, $A in 1-gram – modify tm • what are some words that describe a ||| $N ||| -0.1 • what are some words that describe a ||| $A ||| 0.0 – modify input • add “what are some words that describe a fork?”

7

Entire contents © 2011 Jason S. Chang. All rights reserved.

13年5月27日星期一