(Re)Searching with Google Looking for the core of expertise

Daniel M. Russell Google Search Quality & User Happiness November, 2008 1

Search changes everything

• Your experience of this talk is radically changed from what it was just a few years ago. • Now if I say something like…

“…this topic is much more interesting if you first have a bumper of Nottingham ale…” 2

Nottingham Ale

Search changes the discussion

• … but only if you are facile at search

• KEYPOINT: Of course Google will continue to improve search… but we don’t expect any mindreading modules anytime soon (what IS your intent).

The deep, dark web secret: There will always be information in places and styles that Google won’t be able to crawl. Finally, the interpretation of that data (authority, context) will always be up to the searcher—and searchers will need to understand how to do that. Matlab story

4

► What is expertise? High jump record by year

5

Fosbury flop

Expertise vs. Skill

– Expertise is the set of skills + knowledge that lets you perform at a highly successful level wrt the goals and tasks of that domain – Skill is the performance behavior, often automatic (or nearly so), that is driven by perception and knowledge – Skill level definition varies tremendously over time

7

Expertise

Definition: superior performance of skill(s) due to instruction and extended practice. Medicine, sports, games, work tasks, etc. – Expertise in a domain is usually limited to that domain. (Djakow, Petrowski & Rudik, 1927) – IQ and general intelligence measures are non-predictors of expertise. (Taylor, 1975) – Expertise seems to be a function of pattern knowledge on the domain; including key task behaviors. (Simon & Chase, 1973) – Role of context, place, situation is large. – Expertise acquisition is not a function of time-on-task, but more a function of deliberate-practice-time. (Ericsson,1996) 8

What does it mean to be an expert Google user?

• Music search video…

9

How expert does one become?

• Individuals who start as active professionals or as beginners in a domain change their behavior and increase their performance for a limited time until they reach an acceptable level Beyond this point, however, further improvements appear to be unpredictable and the number of years of … experience in a domain is a poor predictor of attained performance (Ericsson & Lehmann, 1996)

10

Improvements in expertise

expertise level

time-on-task / experience

11

Decline over time without recurring practice

12

► Are you an expert Google user?

• A modest quiz…

13

What do you know what is on the web? 1. Can you find a picture of Barrack Obama on the web? 2. Can you find a picture of Rula Lenska? 3. Can you find a picture of your grandmother? 4. Can you find a picture of MY grandmother? 5. Can you find a video of Ted Williams hitting a home run? 6. Can you find a video of my grandmother hitting a home run? 14

Q1

• In pre-Victorian times, people would often carry a book with them to copy down edifying passages of texts and capture quotations that they’d heard. What is the proper term for this kind of book? A: “commonplace book” Strategy: find a reverse dictionary, [ reverse dictionary ] THEN do [ notebook quotation ] 15

Q2

• I like to read a copy of the book “Chasing the Sun” by Jonathon Green. But I really don’t want to buy it. Where is the closest public library that has this book?

A: Sunnyvale Public Library Strategy: go to Books.Google.com and search for [ chasing the sun ] THEN click on “find this book in a library” 16

Q3

• What zip code has the lowest density of residents?

A: Exercise for the reader…. But a strategic hint: find a data table that has population-by-zipcode.

17

When experts fail: story 1

18

When experts fail: story 2

[ small note chant notation ]

19

How variable is good search performance?

• If even good searchers fail, how can we measure search expertise? • If performance is highly variable… can you really identify “expert” performers? – What causes variability? • What is the variability of Robin Lopez shooting free-throws? Some times great, other times.. .not so great.

20

Hallmarks of an expert Google searcher?

• • • • •

Time-to-first-click? Time-to-result? Ability to handle multiple searches? Ability to manage long-term searches? Use of multiple tabs / multiple windows for coordinated search behavior (to preserve state)?

• Aula (2005): – – – –

Speed of query iteration Query length Fraction of precise queries Speed of evaluating results

21

► Typical searcher behaviors

Weekly Statistics (average user)

Avg #Visits per user

3.9

Avg #Searches per user

9.4

Avg #Search clicks per user

11.2

Avg Visit duration

4 min

Query refinement rate

29%

Next page click rate

10%

Back page click rate

15% 22

• ~60% users have an average of less 1 query/day • Average query length is ~3 words / query • Average visit length is very short (~3 mins)

23

Pew internet results about search

While 66% 92% say they’re search lessinthan confident their 1 time perability day searching

Pew Internet & American Life Project (2005) ±3%

24

Average session duration by query length over time

A. Aula, K. Nordhausen, Modeling successful performance in web searching,, J. ASIST, 57(12):1678-1693 (2006)

25

► Task behavior of experts: Study #1

26

Task type frequency

Count

%

Find Complex

330

34.5%

Find Simple

218

22.8%

Navigate

147

15.4%

Explore/Learn

106

11.1%

73

7.6%

Meta Locate/Acquire

63

6.6%

Play

18

1.9%

350

Number of tasks in sample

Task counts by type

300 250 200 150 100 50 0 FC

FS

N

EL

M

LA

P

Task type

27

Task length frequency

Tasks vary in length, but are most often only 1 or 2 searches long. (Tasks with 1, 2, or 3 searches are 84% of all tasks.) 28

► Difficult searches / difficult tasks

• Classes of difficult searches: – Jokes: Searching for a specific joke for which you don’t remember the punchline – Datasets: – Help manuals: – Tutorials: – Sets of examples:

• How do you become expert at searching for these difficult kinds of content?

29

Study: 392 tasks success vs. failure

• •

Searchers take nearly twice as long on a task before they quit in failure. N = 392, X axis is seconds until S declares success or failure. 30

How do you come up with a question / task for search?

• What happens first? (Chicken or egg.) – The goal? – The query? – The task?

• How is it that the searcher comes up with a question that then leads to search engine use? – What is the basis / process of deciding that this in our mind could be resolved by turning to search?

• Possible answer: Some internal or external speech process frames a question or goal in a way that is amenable to search engine use. 31

Speech & priming effects

• We know there are strong priming, cueing, and fixedness effects in language use. • Do these play a role in the way people approach a search engine query?

32

#7 – Keep looking! Think of synonyms!

•

Sometimes you need to keep trying, thinking your problem through in different ways. Usually, if you think about how someone else would describe the thing you’re looking for, that will suggest search terms for you.

• Question: A friend told me that there is an abandoned city in the waters of San Francisco Bay. Is that true? If it IS true, what was the name of the supposed city?

33

#7 – Answer • Yes, it’s true… there IS an abandoned city near what is now Fremont. It was called Drawbridge. • The trick here is to think about other ways of describing an “abandoned city”—don’t just assume that’s the best way to describe it. Try this search: [ ghost town san francisco bay ] The former town of Drawbridge

34

B. TTR as a function of framing

• I heard that there is an abandoned city in the waters of San Francisco bay. Is that true?

• I heard that there is an abandoned town in the waters of San Francisco bay. Is that true?

35

1.

3726 Was there a town on the San Francisco Bay that disappeared into the bay? If so, what was the name of the town?

2.

3724 Is there a ghost town on the San Francisco Bay that was abandoned because it started sinking into the bay? If so, what was the name of the town? 36

► Introducing new features: What do you pay attention to?

What do you notice that’s different here? Do you see the left-hand nav bar? It’s not normally there.

37

Paying attention?

38

Choose a card, any card…

39

40

41

00:12 [ actor most oscars ]

1:15 [ actor most oscars Academy ] 42

00:12 [ actor most oscars ]

1:15 [ actor most oscars Academy ]

00:10 So this is celebrity with most Oscars… 00:11 Actor… ah… most… 00:13 I’m just going to try that…most Oscars… don’t know… 00:19 (reading) “News results for ‘actors most Oscars’ … “ huh.. 00:25 Oh, then that would be currently “Brokeback”… “prior voices”… “truth in Oscar’s relevance”… 00:32 …now I know… 00:35 … you get a lot of weird things..hold on… 00:38 “Are Filipinos ready for gay flicks?” 00:40 How does that have to do with what I just….did...? 00:43 Ummm… 00:44 So that’s where you can get surprised… you’re like, where is this… how does this relate…umm… 00:45 Bond…I would think… 00:46 So I don’t know, it’s interesting… 01:08 Dan: Did you realize you were in the News section? 01:09 Oh, no I didn’t. How did I get that? . . . 01:10 Oooh… no I didn’t. 43

How to be expert user of a UI?

• •

How does one make sense of a user interface? What do you pay attention to? – – – –

What’s interactive? What’s live? What do various actions do? What model does user have of UI? Groups / Functions / Overall operation / Gestalt

44

What do you notice?

45

46

47

48

49

Inattention & Invisible UI elements

50

So… what do we do?

• How can we design the interface & system to be perceivable? • How can we portray the system to be understandable?

• How do we understand what people are doing? • Between inattention and low-signal density…

51

This page frightened people…. We needed to figure out how to make it “friendly.”

New Adv Search

Rapidly scanning the results: Sequence (and resequence) matter Note scan pattern: Page 3:

Result 1 Result 2 Result 3 Result 4 Result 3 Result 2 Result 4 Result 5 Result 6

Q: Why do this? A: What’s learned later influences judgment of earlier content.

n o p q r ❻

54

Skill of reading a SERP (search engine results page)

– How many results are viewed before clicking? – Do users select the first relevant-looking result they see? – How much time is spent viewing results page?

a result title URL abstract (snippet)

55

► Expert search patterns • Search term selection – Choosing your search words “wisely”

[ constellation ] vs. [ planets visible night sky 2008 ] [ red hat ] vs. [ red hat society ]

• Search is often a multi-step process: 1. find or navigate to a good site (“orienteering”) 2. browse for the answer there

[actor most oscars] vs. [oscars] 56

Expert search patterns

• Teleporting – “I wouldn’t use Google for this, I would just go to…”

• Triangulation – draw information from multiple sources and interpolate Example: “how long can you last without food?”

• Strategic approach – search specific kind of site, looking for information mapping

57

Measurable differences between different task types? Informational/Directed/Closed “Find a painting by Georges Seurat called "La Grande Jatte“”

Informational/Locate “Search for a man's watch that is water resistant to 100 meters and under $100”

8 7

250

Event Count Session Time

200

Event C ount

6

150

5 4

100

3 2

50

1 0

0 InfoDC

InfoU Task Type

List

Locate

S e s s io n T im e ( s e c o n d s )

9

Main effect of task type on: ● Event count (Kruskal-Wallis: χ2(3)=368.3; p<.001) ●

and

Session time (Kruskal-Wallis: χ2(3)=368.7; p<.001)

Info Direct-Closed < Info Undirected <= List < Locate 58

To be a strong user…

…you need to have fairly deep knowledge… – – – –

What sites are possible What’s in a given site (what’s likely to be there) Authority of source / site Index structure (time, place, person, …) Î what kinds of searches? – How to read a SERP critically

59

► Mental models

• How DO people think about what a search engine does? – – – –

Completely keyword search? Full-text indexing? Partial-text indexing? Link anchors?

• What DOES one need to know to use search effectively? – – – –

Relevance? Keyword term frequency? Layered index? Spider / crawling? 60

61

Mental models: Roles

1. How does Google look it up? (how can I say what I want?) 2. Predictable behavior

(when I do X, it does Y)

3. What content is indexed? (what can I search for?) 4. How are the results ranked? (why do they come out in this order?) 5. What’s in the index? (what are the different kinds of things to find)

62

Looking for an image

Froogle? Scholar?

WHY??

Looking for an image here… 63

Many ways to ask about a painting… many ways to respond... Query Terms

OneBox

First Google search result

georges seurat "la grande jatte" georges seurat la grande jatte "la grande jatte" la grand jatte george seurat, la grande jatte george seurat "la grande jatte" painting la grand jatte "la grande jatte by georges seurat" ... george seurat la grande jatte georges seurat painting

None None None None None None None None

The Art Institute of Chicago: Art Access The Art Institute of Chicago: Art Access Seurat, A Sunday Afternoon on the Island of La Grande Jatte The Art Institute of Chicago: Art Access WebMuseum: Seurat, Georges The Art Institute of Chicago: Art Access The Art Institute of Chicago: Art Access Sunday Afternoon on the Island of La Grande Jatte Posters by

None None

Webmuseum: Seurat, Georges Webmuseum: Seurat, Georges

la grande jatte

Image

The Art Institute of Chicago: Art Access

la grande jatte georges la grande jatte by georges seurat georges seurat painting grande jatte la grande jatte painting painting la grand jatte seurat

Product Product Product Product Product

The art institute of Chicago Webmuseum: Seurat, Georges The Art Institute of Chicago: Art Access Seurat, A Sunday after noon on the island The Art Institute of Chicago: Art Access

seurat la grande jatte pic la grande jatte by george seurat seurat la grande jatte image

Book Book Book

FlickrBlog Webmuseum: Seurat, Georges Webmuseum: Seurat, Georges

La Grande Jatte by Georges Seurat painting

Scholar

The Art Institute of Chicago: Art Access

64

…with many OneBoxes...

65

Functional models of search

• Point: People by and large do not have functional models of how Google works in their head. They type words… then things happen… – No causal model – Weak predictors for what will happen – Continually surprised… but not overwhelmed! – And this is all okay…

Hendry & Efthimiadis (in press). “Conceptual models for search engines” In: Web Search: Interdisciplinary Perspectives.

66

A model of Google

67

Mental model

• << Cargo cult>>

Cargo cults of New Guinea (post WW2): a religious response to explain new technology and phenomena

Mental model: How can we portray these?

1. Predictable behavior

Can I predict what will happen when I do X?

2. How is content indexed? Is it full-text? How are images indexed?...... 3. How does Google look it up?

Which keywords should I pick?

4. How are the results ranked?

What does the order mean?

5. What’s in the index? What kinds of documents can I search?

69

► A research literacy

• Six kinds of knowledge & skills needed to search:

pure engine technique

information mapping

domain knowledge

search strategy

medical knowledge site: ricoh.com plumbing knowledge “double quotes” …etc… minus (as exclude) plus (include) filetype:pdf intitle:”cheat sheet” … etc …

reverse dictionary knowing when to shift keyword frequencies knowing when to stop contents of domains move from wide to Wikipedia narrow; preserving knowing what’s state; etc… available … etc… 70

Is this believable?

assessment

How does this link to other information I already know? What is the relationship of this information to authoritative sources?

site-specific knowledge

Knowing particular layout and features of a site (metadata sort/filter; variant query mechanisms; location of “print version” alternate view)

71

Six components of expertise

pure engine technique

domain knowledge

assessment

information mapping

search strategy

site-specific knowledge

72

Custom Search Engines

http://www.google.com/cse/ •

A CSE provides a tailored (usually more selective and focused) search experience for the topic of your choice

•

Sample uses: – – –

Custom search for your web site Group together multiple web sites on a common theme Example: search across all Canadian university web sites

73

Finding a custom search engine (CSE)

• http://www.google.com/coop/cse/examples/GooglePicks – Note that the searches can sometimes be a little funky…

[ Google Custom Search Engine ] 74

Alerts • http://www.google.com/alerts • Scan news, groups, web, videos, comprehensive… and generate emails automatically – Use in conjunction with advanced search techniques

[ Google Alerts ] 75

Tools: Toolbar

• http://toolbar.google.com/ • Download extension to browser— – Switch between different corpora easily

[ Google Toolbar ] 76

Tools: Search web history

• Link in upper right corner of browser on home page (or: www.google.com/history )

77

Mashups

•

Combines data from multiple sources into single view

•

To find mashups: [ mashup search ]

•

Example: http://schoolperformancemaps.com/

78

Advanced Search Tips & Tricks

• When to do an image search? – When looking for a picture… obviously – But consider it when you’re stuck – Try: [ resume ] -- but try doing an IMAGE search for resume!

• Oneboxes: – – – – – – –

[ define:moa ] – note: contrast this with [ define moa ] [ movies palo alto ] [ pizza near mountain view ] [ weather mountain view ] phone number and map: [ kepler’s menlo park ] Flight numbers (to track a flight): [ AA102 ] 79

Advanced search techniques

• All search engines have advanced search capabilities. Here are some of the advanced things I use often… site:stanford.edu -- “site restrict” limits results to JUST that site (Try: [ winograd ] vs. [ winograd site:stanford.edu ] or [ moon image ] vs. [ moon image site:nasa.gov ] )

filetype:pdf – “filetype restrict” limits hits to just files of that type (can be pdf, doc, ppt, jpg, gif, xls, etc. (Try: [ sensemaking filetype:PDF ] )

intitle:”American Heritage” – requires that this string be in the TITLE of of the page that’s found (Try: [ museum ] vs. [ museum intitle:”American Heritage” ] )

80

Advanced search technique

• Use double quotes when needed to specify exact phrase match: – [ “Daniel Russell” ] vs. [ Daniel Russell ]

• OR – [ vacation london OR paris ] – [ wildflower (seeds OR plants) ]

• Number range – [ DVD player $50..$200 ] – [ Google price 2000..2004 ] – [ JFK 1960..1963 ]

81

Advanced search page

•

Access to all of the advanced features w/o having to remember them all!

82

Master Searching Tips

•

Try an image search when normal means fail, you might find something that will be useful or spark your interest in a different way.

•

Word order matters —when it’s not working one way, try another.

•

Don’t leave out the “stop words” when searching for common phrases, (e.g., [ Lord of the Rings] “of the” are NOT stop words)

•

Use double quotes properly, and only when you need them (to find a particular sequence of words) – “Daniel M Russell” or “Palo Alto Library”

83

► Co-evolution

• Search engines will continue to change – change is constant… new document types, new searches, new capabilities… – that’s the point of all our studies / testing – things will continue to change rapidly

84

• Search engines need to match capabilities with user expectations and understandable user mental models Æneed to continually refine understanding of user population’s mental models Æneed to detect when a particular model is in play

85

Shared responsibility

• For search engines: – To create a system that behaves predictably – To understand expectations of entire breadth of users – To help make both beginner & expert use possible

• For our users: – To learn the basics of how search engines work – To have a functional mental model – To continually learn as the research engines evolve 86

END