Sibyl: a system for large scale machine learning Tushar Chandra, Eugene Ie, Kenneth Goldman, Tomas Lloret Llinares, Jim McFadden, Fernando Pereira, Joshua Redstone, Tal Shaked,Yoram Singer

Machine Learning Background Use the past to predict the future Core technology for internet-based prediction tasks Examples of problems that can be solved with machine learning:

• •

Classify email as spam or not Estimate relevance of an impression in context: • Search, advertising, videos, etc. • Rank candidate impressions

The internet adds a scaling challenge:

• •

100s of millions of users interacting every day Good solutions require a mix of theory and systems

Overview of Results Built a large scale machine learning system: • Used recently developed machine learning algorithm • Algorithms have provable convergence & quality guarantees • Solves internet scale problems with reasonable resources • Flexible: various loss functions and regularizations Used numerous well known systems techniques

• MapReduce for scalability • Multiple cores and threads per computer for efficiency • GFS to store lots of data • Compressed column-oriented data format for performance

Inference and Learning

• Objective: draw reliable inferences from all the evidence in our data

• Is this email SPAM? • Is this webpage porn? • Will this user click on that ad?

• Learning: create concise representations of the data to support good inferences

Many, Sparse Features

• • • • •

Many elementary features: words, etc. Most elementary features are infrequent Complex features:

• •

combination of elementary features discretization of real-valued features

Most complex features don’t occur at all We want algorithms that scale well with number of features that are actually present, not with the number of possible features

Supervised Learning

• Given feature-based representation • Feedback through a label: • Good or Bad • Spam or Not-spam • Relevant or Not-relevant • Supervised learning task: • Given training examples, find an accurate model that predicts their labels

Machine learning overview Training data

Label Feature 1,

...

Feature n

Label Feature 1’,

...

Feature n’

Label Feature 1’’,

...

Feature n’’

Machine learning overview Training data

Label Feature 1,

...

Feature n

Label Feature 1’,

...

Feature n’

Label Feature 1’’,

...

Feature n’’

Model

Feature 1 = 0.2,

...

Feature n = -0.5

Machine learning overview Training data

Label Feature 1,

...

Feature n

Label Feature 1’,

...

Feature n’

Label Feature 1’’,

...

Feature n’’

Model

Feature 1 = 0.2,

...

Feature n = -0.5

+ Feature 1’’’,

...

Feature n’’’

Machine learning overview Training data

Label Feature 1,

...

Feature n

Label Feature 1’,

...

Feature n’

Label Feature 1’’,

...

Feature n’’

Model

Feature 1 = 0.2,

...

Feature n = -0.5

+ Feature 1’’’,

...

Predicted label

Feature n’’’

Machine learning overview Training data

Label Feature 1,

...

Feature n

Label Feature 1’,

...

Feature n’

Label Feature 1’’, Label Feature 1’’’,

...

Feature n’’

...

Feature n’’’

Model

Feature 1 = 0.2,

...

Feature n = -0.5

+ Feature 1’’’,

...

Predicted label

Feature n’’’

Example: Spam Prediction • •

• •

Feedback on emails: “Move to Spam” , “Move to Inbox” Lots of features: • Viagra ∈ Document • IP Address of sender is bad • Sender’s domain @google.com • ... Feedback returned daily and grows with time New features appear every day

From Emails to Vectors • User receives an email from an unknown sender Email is tokenized: • ... Viagra ∈ Document Sudafed ∈ Document Find a young wife ∈ Document



...

Compressed instance:

x ∈ {0, 1}n (0, 0, 1, 0, 1, 0, . . . , 0, 0, 1, 0)

From Emails to Vectors • User receives an email from an unknown sender Email is tokenized: • ... Viagra ∈ Document Sudafed ∈ Document Find a young wife ∈ Document



...

Compressed instance:

x ∈ {0, 1}n (0, 0, 1, 0, 1, 0, . . . , 0, 0, 1, 0)

Prediction Models Captures importance of features Viagra ∈ Document => score +2.0 Sudafed ∈ Document => score +0.5 Sender’s domain @google.com => score -1.0

Represented as a vector of weights w = (0, 0, 2.0, -0.1, 0.5, ..., -1.0, ...)

Scoring the email w.x = 2.0 + 0.5 - 1.0

Logistic regression (used for probability predictions) Probability =

Prediction Models Captures importance of features Viagra ∈ Document => score +2.0 Sudafed ∈ Document => score +0.5 Sender’s domain @google.com => score -1.0

Represented as a vector of weights w = (0, 0, 2.0, -0.1, 0.5, ..., -1.0, ...)

Scoring the email w.x = 2.0 + 0.5 - 1.0

Logistic regression (used for probability predictions) Probability =

Parallel Boosting (Collins, Schapire, Singer 2001) • •

Iterative algorithm, each iteration improves model



Updates correlated with gradients, but not a gradient algorithm



Self-tuned step size, large when instances are sparse

Number of iterations to get within � of the optimum: log(m)/�2

Boosting: Illustration

Boosting: Illustration

Boosting: Illustration

Parallel Boosting Algorithm instances

features

µ+ j q(i) =

1 1 + exp(yi (w · xi ))

=



q(i)

i:yi =1∧xij =1

µ− j

=



q(i)

i:yi =−1∧xij =1

wj

+ = η log



µ+ j µ− j



Parallel Boosting Algorithm instances

features

µ+ j q(i) =

mistake probability

1 1 + exp(yi (w · xi ))

=



q(i)

i:yi =1∧xij =1

µ− j

=



q(i)

i:yi =−1∧xij =1

wj

+ = η log



µ+ j µ− j



Parallel Boosting Algorithm instances

features

positive correlation µ+ j q(i) =

mistake probability

1 1 + exp(yi (w · xi ))

=



q(i)

i:yi =1∧xij =1

µ− j

=



q(i)

i:yi =−1∧xij =1

wj

+ = η log



µ+ j µ− j



Parallel Boosting Algorithm instances

features

positive correlation µ+ j q(i) =

mistake probability

1 1 + exp(yi (w · xi ))

=

negative correlation �

q(i)

i:yi =1∧xij =1

µ− j

=



q(i)

i:yi =−1∧xij =1

wj

+ = η log



µ+ j µ− j



Parallel Boosting Algorithm instances

features

positive correlation µ+ j q(i) =

mistake probability

1 1 + exp(yi (w · xi ))

=

negative correlation �

q(i)

i:yi =1∧xij =1

µ− j

=



q(i)

i:yi =−1∧xij =1

wj

+ = η log



step size

µ+ j µ− j



Properties of parallel boosting



Embarrassingly parallel: 1. Computes feature correlations for each example in parallel 2. Feature are updated in parallel







We need to “shuffle” the outputs of Step 1 for Step 2

Step size inversely proportional to number of active features per example

• •

Not total number of features Good for sparse training data

Needs some form of regularization

Learning w/ L1 Regularization

Learning w/ L1 Regularization

Learning w/ L1 Regularization 680

Loss + Regularization

660

640

620

600

580

560

540

520

10

20

30

40

50

Iterations

60

70

80

90

100

Implementing Parallel Boosting + Embarrassingly parallel + Stateless, so robust to transient data errors + Each model is consistent, sequence of models for debugging - 10-50 iterations to converge Data

Model i+1

Model i

MapReduce

MapReduce

Some observations

• We typically train multiple models • To explore different types of features • Don’t read unnecessary features • To explore different levels of regularization • Amortize fixed costs across similar models • Computers have lots of RAM • Store the model and training stats in RAM at each worker • Computers have lots of cores • Design for multi-core • Training data is highly compressible

Design principle: use column-oriented data store

• Column for each field • Each learner only reads relevant columns • Benefits • Learners read much less data • Efficient to transform fields • Data compresses better

Design principle: use model sets

• Train multiple similar models together • Benefit: amortize fixed costs across models • Cost of reading training data • Cost of transforming data • Downsides • Need more RAM • Shuffle more data

Design principle: “Integerize” features

• •

Each column has its own dense integer space

• •

Variable-length encoding of integers

Encode features in decreasing order of frequency Benefits:

• •

Training data compression Store in-memory model and statistics as arrays rather than hash tables



Compact, faster, less data to shuffle

Design principle: store model and stats in RAM

• Each worker keeps in RAM • A copy of the previous model • Learning statistics for its training data • Boosting requires O(10 bytes) per feature • Possible to handle billions of features

Design principle: optimize for multi-core

• Share model across cores • MapReduce optimizations • Multi-shard combiners • Share training statistics across cores

Design principle: use combiners to limit communication

+ Standard Mapper

Mapper with Combiner

Design principle: use combiners to limit communication

• Fewer large shards mean less shuffling, but Map Shard Output to Shuffle

possible stragglers when shards fail Less shuffling

Faster recovery

Map Shard Input Size

Design principle: use combiners to limit communication

• Solution: Multishard Combining • Multiple threads per worker • Many small map shards per thread • One accumulator shared across threads • One supershard per worker... less shuffling • Spread shards from failed workers across the remaining workers ... fewer stragglers

Design principle: use combiners to limit communication

+ Mapper Standard with Mapper Combiner

+++

+

Combiner per Map Thread

Multishard Combiner

Compression results •



Data Set 1



3.2x compression (source is unsorted and has medium compression)



2.6x compression (source is sorted and has medium compression)



1.7x compression (source is sorted and has max compression)



string -> int map overhead < 0.5%

Data Set 2

• •

1.8x compression (default compression options) string -> int map overhead < 0.5%

Performance results Number of models in model set

1 Cores

80 160 240 320 400

2

3

4

5

1.8M 4.0M 4.4M 5.4M 4.5M 1.3M 2.4M 3.0M 4.4M 3.5M 1.4M 2.2M 3.0M 3.9M 3.5M 1.2M 2.0M 2.4M 2.9M 3.3M 1.1M 1.7M 2.4M 2.1M 2.7M

Measurements in features/second per core

Infrastructure challenges Sibyl is an HPC workload running on infrastructure designed for the web

• • • • •

Rapidly opens lots of files



GFS master overload

Concurrently reads 100s of files per machine

• •

Cluster cross-sectional bandwidth overload Denial of service for co-resident processes

Random accesses into large vectors

• •

Prefetch performance Page-table performance

MapReduce challenges



Multi-shard combiners, column-oriented format

Column oriented data format creates lots of small files



Outside the GFS sweet spot

Natural Language Processing Research - Research at Google

Used numerous well known systems techniques. • MapReduce for scalability. • Multiple cores and threads per computer for efficiency. • GFS to store lots of data.

3MB Sizes 3 Downloads 493 Views

Recommend Documents

Natural Language Processing (almost) from ... - Research at Google
Now available at http://trec.nist.gov/data/reuters/reuters.html. 17 .... this case A; for every pair of members Ai, Aj of that word class we ask how the sentence ...

Efficient Natural Language Response ... - Research at Google
ceived email is run through the triggering model that decides whether suggestions should be given. Response selection searches the response set for good sug ...

Speech and Natural Language - Research at Google
Apr 16, 2013 - clearly set user expectation by existing text app. (proverbial ... develop with the users in the loop to get data, and set/understand user ...

K2Q: Generating Natural Language Questions ... - Research at Google
Nov 8, 2011 - Xiance Si. Google Inc. ... however, the keyword paradigm simply does not work. .... These operations do not take grammar into consid- eration ...

Multilingual Language Processing From Bytes - Research at Google
training datasets (no external data sources). .... our model; Section 4 gives training details includ- ing a new ..... Recent results with sequence-to-sequence auto-.

natural language processing
In AI, more attention has been paid ... the AI area of knowledge representation via the study of ... McTear (http://www.infj.ulst.ac.uk/ cbdg23/dialsite.html).

Blunsom - Natural Language Processing Language Modelling and ...
Download. Connect more apps. ... Blunsom - Natural Language Processing Language Modelling and Machine Translation - DLSS 2017.pdf. Blunsom - Natural ...

LANGUAGE MODEL CAPITALIZATION ... - Research at Google
tions, the lack of capitalization of the user's input can add an extra cognitive load on the ... adding to their visual saliency. .... We will call this model the Capitalization LM. The ... rive that “iphone” is rendered as “iPhone” in the Ca

DISTRIBUTED DISCRIMINATIVE LANGUAGE ... - Research at Google
formance after reranking N-best lists of a standard Google voice-search data ..... hypotheses in domain adaptation and generalization,” in Proc. ICASSP, 2006.

EXPLORING LANGUAGE MODELING ... - Research at Google
ended up getting less city-specific data in their models. The city-specific system also includes a semantic stage for inverse text normalization. This stage maps the query variants like “comp usa” and ”comp u s a,” to the most common web- tex

Action Language Hybrid AL - Research at Google
the idea of using a mathematical model of the agent's domain, created using a description in the action language AL [2] to find explanations for unexpected.

AUTOMATIC LANGUAGE IDENTIFICATION IN ... - Research at Google
this case, analysing the contents of the audio or video can be useful for better categorization. ... large-scale data set with 25000 music videos and 25 languages.

DISCRIMINATIVE FEATURES FOR LANGUAGE ... - Research at Google
language recognition system. We train the ... lar approach to language recognition has been the MAP-SVM method [1] [2] ... turned into a linear classifier computing score dl(u) for utter- ance u in ... the error rate on a development set. The first .

Continuous Space Discriminative Language ... - Research at Google
confusion sets, and then discriminative training will learn to separate the ... quires in each iteration identifying the best hypothesisˆW ac- cording the current model. .... n-gram language modeling,” Computer Speech and Lan- guage, vol. 21, pp.

Language-independent Compound Splitting ... - Research at Google
trained using a support vector machine classifier. Al- fonseca et al. ..... 213M 42,365. 44,559 70,666 .... In A. Gelbukh, editor, Lecture Notes in Computer Sci-.

Natural Language Processing (almost) from Scratch - CiteSeerX
Looking at all submitted systems reported on each CoNLL challenge website ..... Figure 4: Charniak parse tree for the sentence “The luxury auto maker last year ...

Distributed Large-scale Natural Graph ... - Research at Google
Natural graphs, such as social networks, email graphs, or instant messaging ... cated values in order to perform most of the computation ... On a graph of 200 million vertices and 10 billion edges, de- ... to the author's site if the Material is used

Online panel research - Research at Google
Jan 16, 2014 - social research – Vocabulary and Service Requirements,” as “a sample ... using general population panels are found in Chapters 5, 6, 8, 10, and 11 .... Member-get-a-member campaigns (snowballing), which use current panel members

MapReduce: Simplified Data Processing on ... - Research at Google
For example, during one MapReduce operation, network maintenance on a running ..... struction of highly-available networked services. Like. MapReduce ...

Large-scale Incremental Processing Using ... - Research at Google
language (currently C++) and mix calls to the Percola- tor API with .... 23 return true;. 24. } 25. } 26 // Prewrite tries to lock cell w, returning false in case of conflict. 27 ..... set of the servers in a Google data center. .... per hour. At thi

On Rectified Linear Units for Speech Processing - Research at Google
100-dimensional principal components. All layers of our ... model replicas P has been set to 100. ... functions localized in the time-frequency domain, but others are more ... acoustic models using distributed hessian-free optimiza- tion,” in ...

The Space Complexity of Processing XML Twig ... - Research at Google
and Google Haifa Engineering Center. Haifa, Israel. [email protected] ..... which we call basic twig queries. Many existing algo- rithms focus on this type ...

Mathematics at - Research at Google
Index. 1. How Google started. 2. PageRank. 3. Gallery of Mathematics. 4. Questions ... http://www.google.es/intl/es/about/corporate/company/history.html. ○.