USOORE41922E
(19) United States (12) Reissued Patent
(10) Patent Number:
Gough et a]. (54)
(45) Date of Reissued Patent:
METHOD AND APPARATUS FOR
(56)
U.S. PATENT DOCUMENTS 4,555,775 A
Inventors: Michael L. Gough, Ben Lomond, CA
4,686,522 A
* 11/1985
Pike ......................... .. 715/790
8/1987 Hernandez
(US); Joseph J. MacDougald, Saint
(Continued)
Petersburg, FL (US); Gina D. Venolia, Bellevue, WA (US); Thomas S. Gilley, New York, NY (US); Greg M. Robbins,
Issaquah, Jr., Georgetown, WA (US); TXDaniel (US); Abhay J. Hansen, Oswal, Rancho Santa Fe, CA (US)
Nov. 9, 2010
References Cited
PROVIDING TRANSLUCENT IMAGES ON A COMPUTER DISPLAY
(75)
US RE41,922 E
FOREIGN PATENT DOCUMENTS
EP
0026205323 0635779
* 10001
OTHER PUBLICATIONS
(73) Assignee: Apple Inc., Cupertino, CA (US)
IBM Technical Disclosure Bulletin, “Transparent Window Selection”, vol. 30, No. 11,Apr. 1988, pp. 2684270.>X<
(21) Appl.No.: 10/163,748 (22) Filed:
(Continued)
Jun. 5,2002
Primary ExamineriPhu K Nguyen (74) Attorney, Agent, or Firijong, Cabello, Lutsch,
(Under 37 CFR 1.47)
Rutherford & Brucculeri LLP
(57)
ABSTRACT
Related US. Patent Documents
Issued:
Jun. 6, 2000
A method and apparatus is described for producing a trans lucent image over a base image created on the display screen of a computer system by a selected ?rst application program,
Appl. No.: Filed:
08/130,079 Sep. 30, 1993
and conducting image operations either on the base image created by the selected application program with reference
Reissue of:
(64) Patent No.:
6,072,489
to the translucent image produced, or conducting image
US. Applications:
operations on the translucent image with reference to the
(63)
Continuation-in-part of application No. 08/060,572, ?led on
base image of the ?rst application program. The ?rst appli
May 10, 1993, now Pat. No. 5,638,501.
cation program runs on a central processing unit (CPU) of a
Int. Cl. G06T 15/00
computer system to produce a base image, and another
(51)
application program referred to as the overlay program is run
(2006.01)
to produce the translucent image such that portions of the base image which are overlapped by the overlay image are at least partially visible through the translucent image. There is
(52)
US. Cl. ...................... .. 715/803; 715/762; 715/781;
(58)
Field of Classi?cation Search ................ .. 345/418,
also a mechanism for blending the ?rst video data and the second video data to produce a blended image on the screen
345/419, 629; 715/204, 762, 781, 803; 395/1334135,
assembly.
345/629
395/1554158, 160 See application ?le for complete search history.
7 Claims, 34 Drawing Sheets
RAM screen buffer
< Blending Engine --—> {
~_.._______'
190 VRAM screen buffer
RAM overlay screen buffer
US RE41,922 E Page 2
US. PATENT DOCUMENTS 4,783,648 A
OTHER PUBLICATIONS
* 11/1988 Homma 6161. ............ .. 715/794
Anonymous, Method to Allow Uers 10 Select Transparent
4,823,281 A
4/1989 Gough
4,827,253 A
5/1989 Maltz
Color for Windows, Mar. 1993, Research Disclosure, pp. 1e3
4,868,765 A
9/ 1989 Diefendorff
Bier et al., “Toolglass and Magic Lenses: The SeeiThrough
4,914,607 A *
4/1990 Takanashiet a1. ......... .. 715/790
Interface,” 1993, Computer Graphics Proceedings, Annual
4,954,970 A 9/1990 Walker 4,959,803 A * 9/1990 Kiyohara et al. .......... .. 715/791
Conference Series, pp. 73*80. Foley, ]_D_, Van Dam, A,, Feiner, S,K,, Hughes, ],F., Com
4,974,196 4,992,781 A * 11/1990 2/1991 Iwami Iwasakiet a1. .............. .. 715/804
puter 909*910, Graphics; Second Principles Edition, andAddisoniWesley Practice, Publishing
5,119,476 A
*
6/1992 TeXier ...................... .. 715/809
Company, Reading, Massachusetts~
5,124,691 A *
6/1992 Sakamoto et a1. ......... .. 715/799
Hearn, Donald and Baker, M_ Pauline, Computer Graphics,
5’185’808 A
2/1993 COk
5,283,526 A * 12/133431 5:815:
1994, pp. 508*511, Second Edition, Prentice Hall, Inc.,
715/729
EpgleWOOd Chffs’NeW Jefsey'.
5’283’867 A
2/1994 Ayley """"""""""" "
5’307’452 A
4/1994 H ahn
1soniWesley Publishing Company, Read1ng, Massachusetts.
5’313’227 A
5/1994 Aoki
Angel, Edward, Interactive Computer Graphics: A Top
5/1994 Hirose
*Down Approach with OpenGL, 1997, pp. 57*5'8, 214*215,
533133571 A 5,351,067 A
*
5,425,137 A
5,425,141 A 5,463,726 A 5,463,728 A
9/1994 Lumelsky 6161. ......... .. 345/561
4127414>Addls°n7W651ey Longman’Inc-’Readmg’Massa'
6/1995 Mohan
Chusens~
@1995 Gedye
Glassner, Andrew S., Editor, Graphics Gems, 1990, pp.
10/1995 Price * 10/1995 Blahut et a1. .............. .. 345/807
5,467,441 A 5,467,443 A 5,469,540 A
11/1995 Stone 11/1995 1911115911 11/1995 Powers
i
Yince, John, ComputerAmmation, 1992,'pp. 134,314, Add
i
5352;“
535813670 A
12/1996 Bier
5590265 A
12/1996 Nakazawa
5,596,690 A
1/1997 Stone
397*399, Academic Press, Inc., San Diego, California. Hiroshi Ishii and Kazuho Arita, “ClearFace: Translucent
Multiuser Interface for TeamWorkStation,” in ACM Sigchi Bulletin, Oct. 1991, pp. 67*68, vol. 23, No. 4, ACM, New YorkaNewyork~
Douglas C. Engelbart and William K. English, “A Research Center for Augmenting Human Intellect,” AFIPS Confer ence Proceedings of the 1968 Fall Joint Computer Confer ence, Dec. 1968, pp. 3954110, vol. 33, San Francisco, Cali
5,617,114 A
4/1997 Bier
fornia. Reprintedby ThompsonBookCompany, Washington
5,638,501 A * 5,651,107 A *
6/1997 Gough et a1. 345/639 7/1997 Frank et a1. ............... .. 345/768
D.C. Hiroshi Ishii and Kazuho Arita, “ClearFace: Translucent
5,652,851 A
7/1997 Stone
Multiuser Interface for TeamWorkStation,” Proceedings of
i
1331011:
ECSCWA91, Sep. 1991, pp. 1634174, Amsterdam, The
5,818,455 A
10/1998 Stone
,
,
5,831,615 A
5,949,432 A
uX 0n
* ll/l998
Drews et al. .............. .. 345/768
9/ 1999 Gough
-
-
Iglceitllfiréands, Ed1tors L. Bannon, M. Robinson and K. '
* cited by examiner
US. Patent
Nov. 9, 2010
US RE41,922 E
Sheet 1 0f 34
m_o:@0365
QRJ
r\_H535
wkSMN o3.owKmm AIHIJW anyUN m
SEQmmJ
mZOMsezUnm
w“mmvm
— w2m$<02%M 15*5.0T5i
Iii 55$2 3.it‘Q
w“MOm UME
m.5551.0*l0 H mm\ m u W.N
.\mm.
X.Jm“.95 2
ZMOEU/AX
m“
US. Patent
Nov. 9, 2010
Sheet 2 0f34
US RE41,922 E
LAUNCH APPLICATION PROGRAM
Figure 2 APPLICATION
DISPLAYS AN
/44
IMAGE
PROCESS CURSOR
OVERLAY TASK?
TRANS
LUCENT
OPAQUE
REQUEST?
MAKE IMAGE TRANSLUCENT
MAKE IMAGE
OPAQUE
52
US. Patent
Nov. 9, 2010
Sheet 3 0f 34
US RE41,922 E
r¢ File Edit
[11 $1
I
EE
r 1 File
Edll
C]
@\
APP #1
62
70
EMIW
' U I WWW/WWW l3 Q'J
Figure 3b
US. Patent
Nov. 9, 2010
Sheet 4 0f34
’ t Hie Edit
31
US RE41,922 E {17 Q?
APP #1
i
62
’ 1 Fix:
.
Edit
“1
33%
62
Figure 3d
US. Patent
( i File
Nov. 9, 2010
Sheet 5 0f34
US RE41,922 E
Edlt
E]
E] @\
APP #1
62
:ram-u El 2 I WWI/IIIWII/l/M'll/IM/m [I . -.';
Figure 3e r‘Fllo Eu:
we
60"
71A
Figure 3f
US. Patent
Nov. 9, 2010
Sheet 6 0f 34
US RE41,922 E
( ‘ Ha 634
95%“
’1 a» am
$16"
60m 71'
N79
Figure 3i
US. Patent
Nov. 9, 2010
Sheet 7 0f34
US RE41,922 E
(-32,767; -32,767)
[80 OPERATING SYSTEM
(0’0)
SCREEN 8|
w—
80”
OVERLAY
(030') ‘f SCREEN “82
(+32,767; +32%?)
V VRAM
SCREEN
"‘60
85)
20)
Figure 4
US. Patent
Nov. 9, 2010
US RE41,922 E
Sheet 8 0f 34
/50
RECORD REACTIVE OR
NON~REACTIVE STATE
MOVE IMAGE
CONTROL FROM
95
SYSTEM SCREEN TO
OVERLAY SCREEN
l BLEND SYSTEM SCREEN AND
OVERLAY SCREEN
/96
INTO VRAM
DISPLAY CONTENTS OF VRAM ON PHYSICAL
DISPLAY MONITOR
99
Figure 5a
US. Patent
Nov. 9, 2010
Sheet 9 0f 34
Apriicaiion Program
- y 101
New Wmduw Routim:
Frame Rec: Routine
Overlay Shicid Cursor Patch
Shl?d Cum» Routine.
Show (2mm): Routine
- leG
US RE41,922 E
US. Patent
Nov. 9, 2010
Sheet 10 0f34
US RE41,922 E
110
INTERCEPT CALL TO SHIELD CURSOR ROUTINE
STORE COORDINATES OF SHIELD RECTANGLE FOR FUTURE BLENDING OPERATION
FIRST TIME DRAWING TO
OVERLAY?
123
124
CREATE OVERLAY
I], 126
r
1 COPY VRAM SCREEN TO RAM
SCREEN BUFFER
REDIRECT FUTURE
PAgngS/gild
DRAWING OUTPUT
(STEP 107, FIG 5b)
INTENDED
FOR VR AM To RAM
Figure 50
128
US. Patent
Nov. 9, 2010
Sheet 11 0f34
@ “01W
EXECUTE AN APPLICATION
EXECUTE THE OVERLAY UTILITY
|’
132
US RE41,922 E
,
Figure 6a
133
@ Figure 6b DISPLAY AN OVERLAY IMAGE ON THE SCREEN SUCH THAT IMAGES ON THE SCREEN WHICH IT MAY OVERLAP CAN BE SEEN THROUGH THE ovERLAY IMAGE
138
l INTERCEPT SCREEN INPUTS
139
WHICH CONTACT THE
OVERLAY IMAGE & PROCESS THE SCREEN INPUTS
UPDATE ACTIVE APPLICATION
PROGRAM ACCORDING TO PROCESSED SCREEN INPUTS
l 40
US. Patent
Nov. 9, 2010
Sheet 12 0f34
US RE41,922 E
[12
@
/
INTERCEPT SHOWCURSOR ROUTINE CALL
142
RECALL SHIELD RECTANGLE COORDINATES
144
CALL BLENDING ENGINE
146
L
PASS CONTROL TO
SHOWCURSOR ROUTINE CALL
Figure 7
148
US. Patent
Nov. 9, 2010
Sheet 13 0f34
@ 152
US RE41,922 E
n4
DIVlDE SHIELD RECTANGLE. INTO INDIVIDUALLY
154
BLENDABLE UNITS (1-32 PIXELS)
156
RETRIEVE RAM SCREEN BUFFER DATA WITHIN SHIELD RECT ANGLE FOR ONE BLENDABLE UNIT
1 RETRIEVE RAM OVERLAY SCREEN BUFFER WITHIN SHIELD RECTANGLE FOR ONE BLENDABLE UNIT
158
BLEND RETRIEVED DATA l/160
TO FORM BLENDED DATA
l WRITE BLENDED DATA
162
TO VRAM
ALL BLENDABLE UNITS DONE?
PASS CALL TO snow CURSOR ROUTINE
|66
Figure 8
US. Patent
Nov. 9, 2010
Sheet 14 0f34
US RE41,922 E
3F%é é Fzgure8a 4.%_/
:
i;
Fzgure 8b
.
//
F zgure 80
d \ Q a
\ /
.
’7
/
\
$§ 47‘“
§®
/'
\
\ /
%®
Fzgure 8d ,
Figure 8e
Figure 8f
"0“
y
/ $$
\ /
‘7“
US. Patent
Nov. 9, 2010
?gure 9
I /
Application Program
.
New w'indnw'~ ,. HR
,
Roudne
Sheet 15 0f34
System
US RE41,922 E
- ‘72
._.._._n.,.____._..._
x/
‘
2%, Poimcr
*~ ~ i74
‘ ()vcllay Siva-mm“, 203
Table
Task Patch
System. /
_
1
Task
IQZN’WW'W
w, 193
' :m_~‘
mm,
/
'
Overlav
I
Show
'
Blendng
3* -
Cuwo: "“ Engine M
Oveday Shield
W,
'
'
..
,1 “M H:
‘2
Patch
Cuml 11' Each I ,1
imp";
'
\
,
\\
Y
1‘
.
\
'_
\“t
'
Shit‘ideso: Nomi ne,
/ ‘ it
I/
“by SHOW (‘ursm Routine
mo
\\\ \
\~ ‘M'N.
I ..
V 96
US. Patent
Nov. 9, 2010
Sheet 16 0f34
US RE41,922 E
'\—192
RAM screen buffer
,/_M_h“\
g
/ _ I\ \ Blendmg Engme M
\ 4 Mm
W4~
. WW“:
/
4/ 190
" "W
Mi
I96'
“
RAW
, VRAM screen buffer
E
RAM overlay N...screen _ buffer
?gure 10
US. Patent
Nov. 9, 2010
Sheet 17 0f34
® 210
US RE41,922 E
Figure 11
U
186
INT ERCEPT CALL TO SHIELD CURSOR ROUTINE
212
/
STORE COORDINATES OF
SHIELD RECTANGLE FOR
214
FUTURE BLENDING
220
DRAWING TO OVERLAY?
DETERMINE WHICH PAGES OF VRAM ARE TOUCHED
BY OVERLAY DRAWING OPERATION
COPY DATA FROM VRAM TO RAM FOR EACH
222
224
TOUCHED PAGE
1 REDIRECT DRAWING TO RAM FOR EACH TOUCI-IED PAGE
'
/
226
230C
PASS CALL TO SHIELD CURSOR ROUTINE
RECORD BUFFER OVERFLOW ERROR IF ANY
228
US. Patent
Nov. 9, 2010
US RE41,922 E
Sheet 18 0f 34
24”; {in gm1 ”
W1%
UH_| . lU92¢HHH {Ea
4g/ llli
wmm
BamOt/SM.» vmm
asEme @5%635
