Comparative User Study of two See-through Calibration Methods Jens Grubert ∗

Johannes Tuemler †

Ruediger Mecke ‡

Michael Schenk §

Volkswagen AG

Fraunhofer Institute for Factory Operation and Automation IFF Magdeburg

Fraunhofer Institute for Factory Operation and Automation IFF Magdeburg

Fraunhofer Institute for Factory Operation and Automation IFF Magdeburg

A BSTRACT Calibration of optical see-through head-mounted displays as well as measuring the overall accuracy of an Augmented Reality System are challenging tasks. This paper describes a user study comparing the execution time and accuracy of the depth-SPAAM and MPAAM see-through calibration methods. While both methods resulted in comparable accuracy inside the calibrated range, one of them was significantly faster to execute. Index Terms: H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems—Artificial, augmented, and virtual realities; H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems—Evaluation/methodology 1

I NTRODUCTION

Head-worn optical see-through displays are used in different Augmented Reality application areas. They are difficult to calibrate as the combination of virtual and real objects occurs within the human eye and as there is no straightforward way to access this information. For the same reason the accuracy of these systems is difficult to determine. Several approaches for calibrating these displays emerged over time (e.g., [5, 2, 3]). Calibration methods have to make a trade-off between overlay accuracy and execution time. There are offline approaches for see-through calibration, avoiding user interaction (e.g., [2]). However, calibration methods involving user interaction have the benefit to be applicable on the fly, e.g., in environments with high accuracy requirements where frequent recalibration can occur. Evaluating the overlay accuracy for optical see-through headmounted displays (HMDs) was previously done with qualitative and quantitative measurements, e.g., by using a camera and carrying out image-based measurements ([2]). However, it is not guaranteed that the measured displacement in a camera image can be observed in real world use, as potential user errors during calibration are ignored. This paper compares the overlay accuracy and execution time of a Single Point Active Alignment method (SPAAM) variant [5] and the Multiple Point Active Alignment method (MPAAM) [3]. Both methods were designed for fast and accurate calibration of optical see-through HMDs through user interaction. In addition to camerabased evaluations, human factors during see-through calibration are considered. 2

U SER S TUDY

2.1

FOR TWO

C ALIBRATION M ETHODS

Test Setup

The user study compares the overlay accuracy and execution time of MPAAM developed by Grubert et al. [3] with a modified version ∗ e-mail:

[email protected] [email protected] ‡ e-mail: [email protected] § e-mail: [email protected] † e-mail:

Figure 1: Subject specifying the perceived position of a virtual marker at a distance of 120 cm.

of depth-SPAAM introduced by Tang et al. [4]. In depth-SPAAM, users sequentially assign several 2-D overlays to a single 3-D control point, while they change their distance to the control point repeatedly. In the modified depth-SPAAM employed in this study a virtual square of changing size has to be superimposed with a paper marker. Thus an inhomogeneous spatial distribution of the users’ position in regard to the control point is enforced. The MPAAM algorithm uses a set of control points on a calibration jig. For both methods an arrangement of nine control points covers a depth range of approximately 60 cm. A camera based inside-out system was employed for tracking. Microvision’s Nomad-ND2100 was used as optical see-through HMD for both conditions. For measuring overlay accuracy users indicate the corner location of virtual markers, that are superimposed to real markers on a test tablet, in different distances. Five markers are arranged on the test tablet, to be visible in the corners and center of the optical seethrough HMD (See Figure 1). Subjects could rest their heads on a chin support. 2.2

Participants

Twenty-two subjects (two females, twenty males) participated in the user study. Their average age was 25.1 (σ 2.32). The subjects were mainly students majoring in computer science. Eight subjects wore corrective lenses (contacts or glasses). Every subject had computer experience. Nine had not any experience with AR. Eleven had never used an HMD. Two had already used HMD more than twice. 2.3

Test Procedure

During the tests subjects performed both depth-SPAAM and MPAAM seated; the order was counterbalanced. Each condition was initially practiced. Subjects were then instructed not to readjust the optical see-through HMD on their heads and repeated both procedures. The result was one set of calibration parameters for each procedure. Afterwards, the calibration error was determined

on a separate test bench. Measurements were taken on a test tablet at a distance of 120 cm (See Figure 1). The subjects used a laser pointer to indicate corner positions of virtual rectangles on the test tablet. The indicated point was marked by a supervisor and the offset from the vertex of the real marker was measured. In addition to measuring the overlay accuracy in the calibrated volume, overlay accuracy outside the range was investigated. To this end, a second test tablet at a distance of 70 cm was employed.

Figure 2: Calibration errors and t-test results at 120 cm (top) and 70 cm (bottom) test distances.

2.4

Hypotheses

While executing depth-SPAAM, users are required to change their position relative to a single 3-D control point several times. The more points have to be aligned, the longer the execution time will be. While conducting MPAAM, users have to position themselves in a fixed point relative to a set of 3-D control points once. Therefore, it was assumed that MPAAM can be executed faster than depth-SPAAM (H1). In contrast, aligning a single point should be easier than having to align nine points at once. Therefore, the error introduced by the user should be smaller and the resulting overlay accuracy should be higher for the SPAAM method (H2). 2.5

Results

Only datasets from twenty of twenty-two subjects could be evaluated. Two male participants had to abort the test, one due to heavy headache, the second due to very poor eyesight. The average execution time for MPAAM was 84 seconds (σ 37.2). The average execution time for depth-SPAAM was 154 seconds (σ 35.7), which is significantly slower than the execution time for MPAAM (t-test, t(34) = 4.56, p = 0.00003). The difference in overlay accuracy at a test distance of 120 cm is statistically not significant for each marker position at a significance level of 1% (see Figure 2, top). Outside the calibrated range (test tablet at 70 cm) the calibration errors differed significantly for each of the five marker positions (see Table 2, bottom). Although both test cycles were designed to cover the same calibration range (approximately 100 - 170 cm), an analysis of the subjects’ tracking data revealed this was not the case. The calibration range for MPAAM ranged on average from 106 cm (σ 2.7) to 166 cm (σ 2.8), and for depth-SPAAM from 87 cm (σ 6.6) to 188 cm (σ 20.6). Thus the average distance between the test tablet positioned at 70 cm and the calibration range for MPAAM was more than twice as large as for depth-SPAAM. Hence, extrapolation errors for 3-D input values outside the calibrated range were higher for MPAAM than for depth-SPAAM. Furthermore, some subjects experienced head and neck discomfort after wearing the optical see-through HMD twenty to thirty minutes. This could be traced back to subjects beeing instructed

to avoid readjusting the display on their heads from the start of the calibration procedure until the end of measurements. 3

D ISCUSSION

AND

C ONCLUSION

The user study revealed that both see-through calibration methods presented here make an overlay accuracy of approximately 0.5 cm inside a calibrated range achievable. With MPAAM the calibration was executed in average within 90 seconds, which was significantly faster than executing depth-SPAAM (H1 was confirmed). The overlay accuracy inside the calibrated range was not significantly higher for depth-SPAAM (H2 was not confirmed). While conducting depth-SPAAM, users still achieved a valid calibration result, even if they diverged from predefined positions, as long as the centers of each displayed 2-D and corresponding 3-D points were aligned. For MPAAM there is only one valid position for each user, in which the 2-D and 3-D points are aligned properly. Therefore, with MPAAM the predefined calibration range was complied to more accurate. A higher overlay accuracy or a larger calibrated range might not easily be achieved with the presented methods. More point correspondences would have to be collected by users, which in turn would lengthen and complicate the procedures. We found that collecting nine point correspondences resulted in a good trade-off between overlay accuracy, task complexity and execution time. Since the specification of the position of the virtual rectangles with a laser pointer was time consuming (approximately 10-15 minutes per calibration method), measurements to check overlay accuracy were only taken at two distances in this test. An alternative would be to use monitors of adequate size set up at the different distances, similar to the setup used by Gang et al. [1]. This would allow using a mouse to enter the specified position. This promises a shorter test time and more precise measurements. For applications with high accuracy requirements a new calibration would have to be conducted as soon as the HMD slips on a user’s head, which can be impractical for real world use. To avoid repeatedly conducting see-through calibration procedures during a workflow, methods should be employed to automatically record the movements of the HMD relative to the user’s head and to automatically compensate resulting overlay errors by adjusting the projection parameters. Further investigations are planned to automatically detect and compensate slippage of the optical see-through HMD. ACKNOWLEDGEMENTS This work was supported in part by a grant from the German Federal Ministry of Education and Research (AVILUSplus project, grant no. 01 IM 08 002 A). R EFERENCES [1] L. Gang, N. Rensing, E. Wesstrate, and E. Peli. Registration of an onaxis see-through head-mounted display and camera system. Optical engineering, 44(2):024002.1–024002.7, 2005. [2] S. J. Gilson, A. W. Fitzgibbon, and A. Glennerster. Spatial calibration of an optical see-through head-mounted display. Journal of Neuroscience Methods, 173(1):140–146, 2008. [3] J. Grubert, J. Tuemler, and R. Mecke. Untersuchungen zur optimierung der see-through-kalibrierung fuer mobile augmented reality assistenzsysteme. In M. Schenk, editor, 6./7. IFF Kolloquium: Forschung vernetzen - Innovationen beschleunigen. Fraunhofer Institut fuer Fabrikbetrieb und -automatisierung, 2009. [4] A. Tang, J. Zhou, and C. Owen. Evaluation of calibration procedures for optical see-through head-mounted displays. In ISMAR ’03: Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality, page 161, Washington, DC, USA, 2003. IEEE Computer Society. [5] M. Tuceryan, Y. Genc, and N. Navab. Single-point active alignment method (spaam) for optical see-through hmd calibration for augmented reality. Presence: Teleoper. Virtual Environ., 11(3):259–276, 2002.