The Bionic Tester

Harry Robinson CAST 2007

The Question Addressed in this Talk

Can test automation improve your exploratory testing?

A Famous Cyborg

“… a creature which is a mixture of organic and mechanical parts. Generally, the aim is to add to or enhance the abilities of an organism by using technology. ” © DC Comics

Some Maybe-Famous Cyborgs

Your Face Here

“Metaphoric Cyborg: … the user locked in a ‘cybernetic circuit’ with their computer terminal” Katherine Hayles, How We Became Posthuman

Are You a Productive Cyborg? It depends on how well you are using your “mechanical parts”

Lame automation

Newfangled stuff

Two Examples of Cyborg Testing Google Talk Multi-User Chat

Google Maps Driving Directions

Google Talk’s Multi-User Chat #2 #3 #4 #5 me

Angela Jacob

#1

#6

MUC is Governed by a Small Set of Rules • Anyone logged in can create a chat room • Anyone in a chat room can invite others into it • A person can join a chat room if invited • A chat room disappears if everyone in it leaves •…

Now consider how you would test this system manually. 8

Scenario 1 Angela logs in

Jacob logs in

Angela creates a room

Angela invites Jacob

Jacob accepts 9

Scenario 2 Angela logs in

Jacob logs in

Angela creates a room

Angela invites Jacob

Jacob declines 10

Scenario 3 Angela logs in

Jacob logs in

Jacob creates a room

Jacob invites Angela

Angela accepts 11

Scenario 4 Both log in

Each creates a room

Each invites the other

Both accept

Angela logs out 12

Scenario 5 Both log in

Each creates a room

Each invites the other

Jacob accepts

Jacob logs out 13

Hold it! We are barely 5 test cases in, and already: • I’m already losing track of what we’ve covered • I’m starting to make reckless assumptions • I haven’t even reached a 3-person scenario yet

14

!

Here’s What the Rules Look Like (version 1)

* 15

Here’s What the Rules Look Like (version 2)

* 16

Here’s What the Rules Look Like (version 3)

* 17

The Machine Can Now Generate Test Sequences

* 18

1. 2. 3. 4. 5. 6.

Angela logs in Jacob logs in Angela creates a room Angela invites Jacob Jacob accepts Jacob logs out

What Do We Have to Track?

• Who is logged in • Who is invited into a room • Who is in which room

Bonus At any point we are able to oracle the state of the system.

19

What’s Exploratory About This? 1. Creating the model forced a systematic exploration of the behavior.

2. You can start with a small model and build in the direction you think will do the most good.

* *

*

20

3. You can choose where to “steer” the tests.

* 21

1. 2. 3. 4. 5. 6. 7.

Angela logs in Angela logs out Jacob logs in Jacob logs out Angela logs in Jacob logs in Angela logs out

Welcome to the Hotel California …

* Disable arbitrary logouts to force scenarios 22

Or, Welcome to the Hotel Las Vegas

* Penalize arbitrary logouts to ‘encourage’ scenarios 23

You Can Let the Machine Shorten the Repro

_

+

1000s of transitions happen

Google Maps Driving Directions

25

Investigating the Human-Machine Boundary

11 miles

1.4 miles

Nope, the Lat/Long Heuristic Misled Us

11 miles

1.4 miles

How About Reverse Directions?

Yes, That Works Very Nicely

Sometimes Very Nicely Indeed

But are “scenic return routes” bugs?

from: 92880 to: 99212

from: 99212 to: 92880

And, if they are bugs, how can a program find them reliably?

from Miami to Bakersfield distance = 2791 miles

from Bakersfield to Miami distance = 2793 miles

Compare the Miami/Bakersfield routes difference = (2793 - 2791) = 2 miles

This difference falls below our “40 miles” threshold. And pixel diff’ing* confirms that the 2 routes are similar. So far, so good. * Much thanks to Sebastián Schiavone for his Eggplant pixel-comparison routine.

from Miami to San Francisco distance = 3072 miles

from San Francisco to Miami distance = 3111 miles

Compare the Miami/San Francisco routes difference = (3111 - 3072) = 39 miles

This difference also falls below our “40 miles” threshold. But pixel diff’ing shows that the 2 routes are very different.

Conclusions Scripted automation has its place, but it shouldn’t be all you use. Exploratory automation extends our reach into an application’s behaviors. Exploratory automation can start with the scripts you already have. Don’t prioritize prematurely. Be prepared to accommodate the machines. Be prepared to be creative.

Acknowledgments Gagan Gupta Anshul Tandon Joel Hynoski Maria Katsova Engin Uzuncaova Lydia Ash

39