Improving Visual Servoing Control with High Speed Cameras Alberto Soria, Rub´en Garrido and Ra´ul Vazquez CINVESTAV-IPN. Departamento de Control Autom´atico Av. I.P.N. 2508, Zacatenco. M´exico, DF. 07300 [email protected] Abstract— In this paper, we present a visual servoing control scheme using a fast digital camera at higher frame rates. It is shown that employing a digital camera at high speed improves closed loop bandwidth compared with a standard analog camera. Moreover, higher bandwidth allows to add damping at the visual level, thus obtaining performances similar to those obtained with joint damping.

I. I NTRODUCTION

Visual control of robots allows for non-contact measurements of the environment as opposed to the traditional encoder and end limit switches. It also allows to deal with problems associated with backlash, flexibilities or partially known robot kinematics as for example in the programming of automatic robotic assembly work cells [11]. The task in visual control of robots is to use visual information to control the robot end-effector pose relative to a target object or a set of target features. The two most important Charged-Coupled Device (CCD) camera technologies used for visual servoing are analog and digital CCD cameras. An important part of the research in visual servoing is done using analog cameras that conform to the RS170 or CCIR video standards, having a frame rate of 30 Hz and 25 Hz respectively. Analog cameras can be used up to 60 Hz as shown for example in [4] and [6]. On the other hand, the development and availability of fast digital cameras has given place to faster visual servoing applications such as those proposed by [1], [5] and [14], using frame rates of 120 Hz, 995 Hz and 100 Hz frames per second respectively. In this paper, we are interested in studying the closed loop control behaviour of visual servoing algorithms employing analog and digital cameras applied to a planar 2dof robot in a fixed camera configuration. This will allow to compare the characteristics for both types of cameras such as the trade off between speed and cost, jointbased damping or visual based damping. The remainder of the paper is organized as follows. Section II describes the main characteristics of standard analog and digital cameras. Section III will be concerned with the Jacobian transpose control algorithms employed in the experiments. Section IV focuses on the experimental setup as well as the experimental results and finally, a summary of the significance of the paper is provided in section V.

II. C AMERA T ECHNOLOGY A. CCD Technology Electronic cameras used in machine vision applications employ a CCD sensor invented by Boyle and Smith in 1970 [2]. This development allowed the replacement of bulky Vidicon image tubes. A CCD is a highly sensitive photon detector divided into a large number of detecting sites or pixels. Pixels are arranged in grid patterns that are used to build up an image of the scene of interest. A photon of light which falls within the area defined by one of the pixels will be converted into one (or more) electrons. The charge, or number of electrons collected, will be directly proportional to the intensity of the scene at each pixel. The charge is then transferred to a readout register. The end of the readout register is an amplifier which measures the value of each charge and converts it into a voltage so that the scene can be reconstructed. 1) Analog Cameras: In standard Analog cameras a full image (or frame) is transmitted in two pieces: first, the odd rows (or odd field) are transmitted followed by the even rows (or even field). This technique is referred as interlaced imaging, and was developed as part of television broadcast standards to improve the illusion of motion within the available signal bandwidth. The most common standards for monochrome video are the CCIR and the RS-170 used in Europe and North America respectively. Frame rate is 30 Hz for the RS-170 and 25 Hz for the CCIR standard. In order to achieve higher frame rates using analog cameras, a single field capture (odd or even) is configured as a frame capture. Since the number of fields is two times the number of complete frames, the frame rate (or field rate when using a single field) can be incremented up to 60 Hz. However, odd and even field pixels corresponding to vertically disparate points in the scene can cause problems with target velocity estimators [4]. In an analog CCD camera, its electronics are used to obtain the output gray level voltages values for each pixel and a digitizer of frame grabber must be used to digitize the image. The task of the frame grabber is to assemble a digital image for processing in the computer from the video signal transmitted by the camera. First, the frame grabber samples the incoming analog video signal. The sampling intervals are given by the pixel clock, which can be generated internally in the frame grabber or transmitted from the camera. The sample voltage values are then converted

into a numerical value by an A/D converter. These values are then stored until a complete image has been put together. Two main modes can be distinguished: continuous and sampled. In continuos mode the frame grabber overwrites the current image in its memory area as soon as a new image arrives thus the image changes constantly. This mode is usually used to adjust exposure and focus without an additional video monitor. Acquisition mode uses the vertical synchronization signal to start de next image. The acquisition then stops with the image so that the image processing software can read it. Analog camera synchronisation with external events can be problematic, for example, when a frame grabber receives a trigger it will capture de next frame of video and cannot be asynchronous with an object moving at high speed, thus having a fixed bandwidth. The bandwidth is also limited by cabling and analog noise. The cost of analog machine vision CCD cameras ranges from US300 to US600. 2) Digital Cameras: Digital cameras use digital transmission methods as the RS-422, RS-644, IEEE-1394 or Camera Link that allow to deal with noise related problems: the RS422 allows data rates below 30 megabits per second (Mbs). The RS-644 Standard (also to known as LVDS; Low Voltage Differential Signaling) lowers the transmission voltages allowing faster switching voltages (up to 400 Mbs) while preserving noise immunity characteristics. The IEEE-1394 also known as Firewire, is a 400 Mbps digital bus system that allows to interface cameras to computers; some of its key features are: multiple cameras connection, multiple computer image capture from one camera and camera control with full plug and play camera upgrades. Care should be taken when selecting IEEE-1394 because many cards are aimed at the consumer digital video market and can be problematic in critical machine vision applications. Camera Link is a digital camera interface that allows higher data transfers speeds using fewer cables with standardized connectors cabling and data protocols. This interfacing uses LVDS technology with serialisation of parallel data. Seven bit groups are transmitted over 5 twisted-pair cables allowing a total data speed of 1.8 gigabits per second [7]. In a digital CCD camera, the analog output voltage values for each pixel are converted into a digital form by the camera electronics. Digital cameras are also referred as progressive scan cameras. A digital interface card is usually employed when using a digital camera. Digital cameras can be asynchronously triggered thus allowing a variable bandwidth. The cost of digital cameras is starts around US600. B. Advantages and disadvantages: Digital vs. Analog The most important advantages of digital cameras are: high frame rate, noise immunity, ease of asynchronous triggering, variable bandwidth, square pixels and ease to perform spatial measurements. The main disadvantages are their cost, a more complicated cabling and the need for more illumination at high frame rates. The two main advantages of analog cameras

are their low cost and well established transmission standards. The disadvantages of analog cameras are: slow frame rate, low noise immunity, difficult asynchronous triggering synchronization, rectangular pixels and a greater difficulty for spatial measurements. As the cost for digital cameras becomes more accessible, the trade off of their cost advantages will become more interesting and will allow more pure-visual oriented applications. Analog cameras however, are still used in machine vision application using vision-joint based algorithms due primarily to their low cost-advantages characteristics. Table I resumes the most important characteristics of digital and analog camera technologies. III. JACOBIAN T RANSPOSE V ISUAL ROBOT C ONTROL Kelly [8] proposed a control algorithm based on the Jacobian transpose controller in a fixed camera configuration. From this algorithm, two other Jacobian transpose algorithms were derived and are similar to those proposed by [3] (in the sequel referred as joint based damping) and [10] (referred later as image based damping) were experimentally tested by Vasquez [13]. The main difference among these last two algorithms is the way that the damping is introduced. The joint-based, damping is introduced by means of the product of joint Jacobian and joint velocities in contrast with the image-based damping that is introduced by means of the image feature velocities. It must be noted that the visual feature velocities will directly depend on speed of the image acquisition system and consequently the bandwidth of the visual-based damping. Dynamics of a two degrees of freedom revolute joint robot in the absence of friction or other disturbances can be described by [12]: M q
q   C q q
q  G q
 τ

(1)

where q is 2 1 vector of joint displacements, τ is the 2 1 vector of applied joint torques, M q
is the 2 2 symmetric positive definite manipulator inertia matrix, C q  q
is the 2

1 vector of centripetal and Coriolis torques and G q
is the 2 1 vector of gravitational torques. kinematics provides end effector position xR   Direct xR  yR T  f q
with respect to the robot coordinate frame in terms of q ℜ2 with f : ℜ2  ℜ2 . Robot Jacobian J q
 ℜ2  2 is defined by J q
 ∂ ∂f  qq [12]. A point xR in the robot coordinate frame can be expressed in the screen coordinate frame using the three following elements: a) a rigid body transformation, b) the lens perspective transformation and c) its conversion to the image frame. It is assumed that quantisation as well as lens distortion is negligible. End-effector position xS and target  xS position in the screen frame are described by:

TABLE I A NALOG AND D IGITAL CAMERA CHARACTERISTICS .

Transmission Standards Frame rate Interfacing Synchronisation Resolution Pixel form Cost

Digital RS-422,RS-644,IEEE-1394,Camera Link 1KHz Digital interface card Easy Low, medium and high Square $600



xS  α hR θ
xR
OR  CS   xS  α hR θ
xR
OR  CS (2) where α is the scale factor, h the magnification factor, R θ
is the clockwise rotation matrix, OR is the intersection between camera optical axis and the robot workspace and CS the image centre. The image position error can be defined by the visual distance between the end-effector and target  positions as x S  xS
xS which can be written using (2) as:   xS  α hR θ
f q
d
f q
(3) Supposing that joint data q, q , are available and h, OR are unknown, the control problem is to find a control law τ such that the robot end effector reaches, in the image supplied on the screen, the target point in the robots workspace, i.e.  lim x  0 ℜ2 . A control law that solves the control t ∞ problem [8] using the former robot control description is written as: 

τ  α hJ q
T K p f q


d


f q



Kv q  G q


(4)

This last control law cannot be implemented because f q
d and f q
are unknown. This is solved using (3) and expressing the unknown vector α h f q
d
f q
in terms of  the image position error xS to obtain the law proposed in [9]:  τ  J q
T K p R θ
T x s
Kv q  G q
(5) To obtain the first of the control law that will be used later, a differentiation with respect to time is made of (3) and using the Jacobian definition gives the joint based damping control law:

τJB  J q
T R θ


T





KP xS
KV Kh J q
q   G q
JB

JB

(6)

with Kh  α h  R θ
. The imaged-based damping control law is obtained by   d x differentiating x s  xS
xS , i.e., dtS 
x S , and is then given by:

τIB  J q
T R θ


Analog RS-170,CCIR 60Hz Frame grabber Difficult Low to medium Rectangular $300-$600

T

 

KP x S
KV x S   G q
IB

IB

(7)

The proportional gain matrixes are KP and KP , whereas JB IB the derivate gains are given by KV and KV for each one JB IB of the control algorithms. IV. E XPERIMENTAL R ESULTS A. Experimental Setup A rapid prototyping architecture based on standard off the shelf hardware and software was employed to carry out the experiments, details for this architecture can be found in [6]. The experimental set up consists of a in-house made 2dof robot, a development computer, a control computer and a vision computer. We employed a Ultra Ni-Vision DC12V and a Dalsa CAD1 Digital camera together with a 1408 and 1422 National Instruments cards respecteively. The visual sampling period for the slow camera experiments is 60 Hz and for the fast camera experiments is 600 Hz. The control sampling period is 0.7 ms. The resolution and pixel size for the analog and digital cameras are different. In order to compare the performances with respect to a common base, we calculated a scaling factor for the analog camera pixels. This factor converts the analog camera pixels information to the digital camera pixels equivalent, thus allowing the same physical displacement of the end effector for the same reference when using the analog and digital cameras In order to implement the joint-based control (6) we use a high-pass filter (8) in order to calculate an estimate of q from joint measurements. fc s (8) s  fc Damping for the joint-based control law is introduced using KV and q,  thus depending on the optical encoder JB data that is sampled at 0.7 ms. The the cut-off frequence for the high-pass filter is then f c =2243 rad/sec ( f c =357Hz for a sampling frequency of 1428Hz (0.7ms)). For the image-based control law (7) we also use the highpass filter (8) in order to calculate an estimate of x S from the

TABLE II D IGITAL AND A NALOG C AMERA X

Camera Digital Analog

Algorithm Image Image

Frequency

Filter

60 60

Digital Analog

Image Image

600 60

Digital Analog

Joint Joint

60 60

Digital Analog

Joint Joint

600 60

94s s 94 94s s 94 940s s 940 94s s 94 2243s s 2243 2243s s 2243 2243s s 2243 2243s s 2243

Algorithm Image Image

Frequency

Filter

60 60

Digital Analog

Image Image

600 60

Digital Analog

Joint Joint

60 60

Digital Analog

Joint Joint

600 60

94s s 94 94s s 94 940s s 940 94s s 94 2243s s 2243 2243s s 2243 2243s s 2243 2243s s 2243

KP2 0.12 0.048

KV 1 0.02 0.009

KV 2 0.02 0.016

Rise Time

Bandwidth

0.28sec. 0.35sec

1.000Hz 0.606Hz

0.15 0.046

0.12 0.048

0.017 0.009

0.022 0.016

0.25sec. 0.35sec.

1.321Hz 0.606Hz

0.7 0.46

0.7 0.34

0.09 0.08

0.14 0.14

0.22sec. 0.29sec.

1.448Hz 1.176Hz

1 0.46

0.8 0.34

0.12 0.08

0.12 0.14

0.20sec. 0.29sec.

2.008Hz 1.176Hz

AXIS

M OVEMENT.

KP1 0.08 0.057

KP2 0.08 0.055

KV 1 0.012 0.012

KV 2 0.013 0.013

Rise Time

Bandwidth

0.30sec. 0.35sec.

0.976Hz 0.428Hz

0.11 0.057

0.114 0.055

0.013 0.012

0.0125 0.013

0.29sec. 0.35sec.

1.046Hz 0.428Hz

0.75 0.6

0.7 0.6

0.12 0.1

0.12 0.13

0.27sec. 0.31sec.

1.376Hz 0.804Hz

1.8 0.6

0.8 0.6

0.06 0.1

0.12 0.13

0.23sec. 0.31sec.

1.738Hz 0.804Hz

visual data from the camera. Damping for the image-based control law is introduced using KV and x S , thus depening on IB the camera data that is sampled at 16 ms and 1.6 ms. The the cut-off frequence for the high-pass filter for the image data sampled at 16ms is f c =94 rad/sec ( fc =15Hz for a sampling frequency of 60Hz (16ms)) and for the image data sampled at 1.6ms the values of the filter are f c =940 rad/sec ( f c =150Hz for a sampling frequency of 600Hz (1.6ms)). In all Experiments in the X and Y axis, we use a square wave reference of 30 15 pixels at a frequency of 0.2Hz. A frequency response of the robot was obtained in the form of bode plots to calculate the bandwidth(BW in Hz) for each experiment. The results also present the rise time (RS in sec) for each algorithm under test. To study the bandwidth behaviour using the two damping control laws presented earlier, we propose two sets of experiments, one for the digital camera and other for the analog camera. In each experiment we tuned the control law in order to achieve the fastest response possible with minimum overshoot and without vibrations. Tables II and III show the results for the x-axis movements and the y-axis respectevely. The results in each table are grouped by algorithm(image-based, joint-based) and by


M OVEMENT.

KP1 0.14 0.046

TABLE III D IGITAL AND A NALOG C AMERA Y

Camera Digital Analog

AXIS

image sampling frequency(60Hz and 600Hz). Figures 16 show the x-axis movement for a step reference change corresponding to the the data presented in table II. The yaxis movement is not presented due to space limitations. B. X-Axis Movement. The digital camera using the image-based algorithm, at a sampling frequency of 60Hz has a higher bandwith of 1.000Hz and a shorter rise time of 0.28sec. than the analog camera having a bandwith of 0.606 with a rise time of 0.35sec. This higher bandwidth for the digital camera is explained by the fact that the digital camera has better noise inmunity properties than the analog camera. This better inmunity introduces less noise in the control loop allowing higher derivative and proportional gains. When the digital camera is sampled at 600Hz allows, as expected, a higher bandwith of 1.321Hz with a rise time of 0.25sec. compared with the analog camera bandwith of 0.606 and a rise time of 0.35sec. The higher bandwidth is primarilly due to the higher frame rate and the digital camera noise inmunity properties mentioned earlier. It should be clear that the digital camera sampled at 600Hz has a higher bandwidth than when it is sampled at 60Hz. With the joint-based algorithm, the digital camera exihibits in general similar results as above, allowing higher bandwidth

values than the analog camera, 1.448Hz vs. 1.176Hz and 2.008Hz vs. 1.176Hz. The rise time was in these cases are 0.22sec. vs. 0.29sec and 0.29 vs. 0.20. When the digital camera uses the joint-based algorithm, higher bandwidth values are obained than with the image-based algorithm. (1.448Hz vs. 1.000Hz, 2.008Hz vs. 1.32Hz). The joint-based results are explained due to the fact that the sampling is much faster (at 0.74ms) and thus allowing a higher badwidth. It is interesting to note that the bandwidth of the digital camera with the image-based algorithm at 600Hz has a higher bandwidth than the analog camera using the joint-based alorithm at 60Hz, 1.321Hz vs. 1.176Hz. This confirms the fact that digital cameras allow a comparable visual damping than the analog camera with joint encoder based damping, thus obtaining comparable performances with a pure visual servoing control sheme. C. Y-Axis Movement. In table III the obtained values follow in general the same type of behavior as for the x-axis movement with the difference that the obtained values are smaller. This is due to the fact that when performing the y-axis movement the gravity vector has different values for the used references and that the gravity compensation used does not compensate sufficently for this movement. V. C ONCLUSIONS The use of the digital camera allows higher bandwidth and smaller rise time. The image-based control using the digital camera is comparable to the joint-based control using the analog camera, hence digital camera technology has better performances than its analog counterpart. The best results obtained with the joint-based control law is due to the fact that the damping term is based on the encoder information sampled at a faster rate (0.7ms). This allows higher bandwidth than the image-based control law. However, as the joint-based control law requires encoder information it is not a pure visual servoing scheme. The results show that the digital camera has a better noise inmunity than the analog counterpart when comparing the cameras perfonce at 60Hz and 600hz with the joint-based and image-based control laws for both X,Y axis movements. At the present, the cost for digital camera technology is still above analog cameras. As digital technology prices become more accesible applications involving visual servoing control will adopt digital techology due to the superior performances. VI. REFERENCES [1] Gangloff, Jaques.- Asservissements visuels rapides d’un robot manipulateur a` six degr´es de libert´e. PhD. Thesis. Universit´e Louis Pasteur, Strasbourg, France:1999. [2] Boyle, W. and Smith, G. - “Charge coupled semiconductor devices”. Bell Syst. Tech. Journal. Vol 49. 1970. pp. 587-593.

[3] Chea, C. et al.- “Asymptotic stability of robot control with approximate Jacobian matrix and its application to visual Servoing”. Proccedings of the 39th IEEE Conference on Descition and Control. 1994. pp. 39393944. [4] Corke, P.- Visual Control of Robots: high-performance visual servoing. Research Studies Press: Taunton, Somerset, England. 1996. [5] Cowan, N.; Weingarten, J. and Koditschek, D.- “Empirical Validation of a New Visual Servoing Strategy”. Proceedings of the 10th IEEE International Conference on Control Applications. M´exico City, M´exico. 5 -7 Sept. 2001. [6] Garrido, R.; Soria, A.; V´asquez, I. and Vazquez R.“An architecture for Rapid Prototyping of Visual controllers for Electro Mechanical Systems”. Proceedings of the 10th IEEE International Conference on Control Applications. M´exico City, M´exico. 5 -7 Sept. 2001. [7] Greer, C.- “Camera Fundamentals for Machine Vision”. Eye On Imaging. September issue. Laurent, Quebec: Coreco Imaging. 2001. [8] Kelly, R. - “Robust Asymptotically Stable Visual Servoing of Planar Robots”. IEEE Transaction on Robotics and Automation. Vol 12, N . 5. Oct. 1996. pp.759-766. [9] Kelly, R. and M´arques, A. - “Fixed-eyedirect visual feedback control of planar Robots.” Journal of System Engineering. Vol 4 No.5. Nov. 1995. pp.239-248. [10] Lefeber, E. et al.- “Adaptive and Filtered Visual Servoing of Planar Robots”. Preprints of the 4th IFAC Nonlinear Control Systems Design Symposium. Vol. 2, 1998. pp. 563-568. [11] Nelson, B; Papanikolopoulos, N. and Khosla, P.“Visual Servoing For robotic Assembly”. In Visual Servoing-Real-Time Control of Robot Manipulators Based on Visual Sensory Feedback. Hashimoto, K. Editor. World Scientific: New Jersey,1993. pp. 139-164. [12] Spong, M. and Vidyasagar, M.- Robot Dynamicas and Control. New York: John Wiley and Sons, 1989. [13] V´asquez, I.- Control Visual de un robot Planar de dos Grados de Libertad. Msc. Thesis. Departamento de Control Autom´atico. M´exico, 2002. [14] Zergeroglu, E.; Dawson, D.; De Queiroz, M. and Behal A.- “Vision-Based Nonlinear Tracking Controllers With Uncertain Robot-Camera Parameters”. IEEE/ASME Transaction on Mechatronics.Vol. 6, N . 3. Sept. 2001.pp.322-337.

X axis Movement

X axis Movement

50

50 X Position Y Position X Set Point Y SetPoint

X Position Y Position X Set Point Y SetPoint

40

40

30

Pixeles

20

20

10

10

0

0 −10

0

1

2

3

4

5

6

time(sec.) −10

0

1

2

3

4

5

6

time(sec.)

Fig. 1.

Fig. 4.

Joint-based 60Hz Digital Camera Visual Control.

Image-based 60Hz Digital Camera Visual Control. X axis Movement 50 X axis Movement

X Position Y Position X Set Point Y SetPoint

50 X Position Y Position X Set Point Y SetPoint

40

40

30

Pixeles

Pixeles

30

20

20

10 10

0 0

−10 −10

0

1

2

3

4

5

0

1

2

3

4

5

6

time(sec.)

6

time(sec.)

Fig. 5. Fig. 2.

X axis Movement

X axis Movement

50 X Position Y Position X Set Point Y SetPoint

X Position Y Position X Set Point Y SetPoint

40

40

30

Pixeles

30

20

20

10

10

0

0

−10

Joint-based 600Hz Digital Camera Visual Control.

Image-based 600Hz Digital Camera Visual Control.

50

Pixeles

Pixeles

30

−10 0

1

2

3

4

5

6

0

Fig. 3.

Imaged-based 60Hz Analog Camera Visual Control.

1

2

3

4

5

time(sec.)

time(sec.)

Fig. 6.

Joint-based 60Hz Analog Camera Visual Control.

6