Acta Psychologica North-Holland

83

54 (1983) 83-98

HOW A DISCONTINUOUS MECHANISM CAN PRODUCE CONTINUOUS PATTERNS IN TRAJECTORY FORMATION AND HANDWRITING * P. MORASSO,

F.A. MUSSA IVALDI and C. RUGGIERO

Uniuersity of Genoa, Italy Accepted

May 1983

Handwriting is considered in the framework of trajectory formation, i.e. the class of motor activities which have the main purpose of generating kinematic-geometric patterns. A model for the generation of trajectories is proposed, which is based on the notion of stroke and on a method for composing chains of strokes. The input of the model is a (discrete) list of stroke descriptors and the output of the model is the (continuous) motion of the hand. Experimental trajectories were sampled and their geometric, timing, and kinematic aspects were analyzed. The model was then fitted to the data for extracting the stroke parameters (duration, length, symmetry, angular change) and for computing their histograms.

1. Introduction The concept of trajectory formation refers to the activity of the Central Nervous System (CNS) in programming the geometric and kinematic characteristics of the arm movements during the execution of a great variety of tasks, such as pointing, gesturing, writing, drawing, manipulating objects. A fundamental hypothesis on trajectory formation was made by Bernstein (1957) and Lashley (1951), observing that the CNS is likely to conceive of motor programs in terms of spatial coordinates of the hand rather than in terms of muscle and joint patterns, and was * The authors thank Prof. V. Tagliasco for his encouragement and thank Mr. S. Bianchi, Mr. P. Del Carratore, and Mr. S. Giolitto for having implemented the computer program which generates synthetic handwriting. This work was partly supported by a Bilateral Research Program and by the Center of Bioengineering and Anthropomorphic Robotics of the Italian Research Council. Mailing address: P. Morasso, Center of Bioengineering and Anthropomorphic Robotics, Istituto Elettrotecnica, University of Genoa, Viale Causa 13, 16145 Genoa, Italy.

OOOI-6918/83/$3,00

0 1983, Elsevier Science Publishers

B.V. (North-Holland)

subsequently supported by the experimental finding (Morass0 1981; Abend et al. 1982) that for simple movements of pointing the motion of the hand exhibits a stable bell shaped velocity profile whereas the joint angular rotation patterns depend upon the starting point, direction, and amplitude of the movements. If we now extend our investigation of trajectory formation from the simple case of pointing to more articulated cases, like avoiding obstacles or handwriting, we shall have to face the apparent paradox of a process that looks continuous and smooth from the outside but must be discontinuous at some motor cognitive level, i.e. at a level of motor or visuo-motor reasoning. In fact, the smoothness and uniformity of manual movements cannot be trivially attributed only to the filtering action of the mechanical and peripheral characteristics of the motor system (inertia, viscosity, elasticity) because such filtering action is grossly non-linear and, in particular, it depends strongly on posture (because apparent inertia depends strongly on posture (Benati et al. 1980)) and on the levels of contraction of the muscles (which determine the viscous-elastic components of the mechanical transfer function). Furthermore, since the frequency contents of usual movements are likely to be well inside the bandwidth of the mechanical transfer function [l], the smoothness of the movements must have some other main root. The inherent discreteness of complex movements, particularly in handwriting, can be related also to implications in common expressions in language, such as in the use of the word “stroke” to indicate some basic or recognizable motor act (e.g. “a stroke of the pen”). However, the models which have formalized the notion of stroke (Mermelstein and Eden 1964; Crane and Savoie 1977; Herbst and Liu 1977) particularly for the purpose of pattern recognition, define the strokes as segments of the handwritten trace, directly observable from the trajectories. In contrast, the notion of stroke that we propose has different roots and is quite similar to the geometric and timing structure of the pointing movements. Furthermore, we propose that the internal representations of articulated trajectories consist of chains of straight and curved segments (i.e. generalized polygons of strokes) and that the smoothness of the script (in contrast with the discontinuity of the [l]

The mechanical transfer function of a limb depends on posture and on muscle activity, but for usual movements an educated guess is close to the 10 Hz figure. See for example the data of Agarwal and Gottlieb (1977) and of Cannon and Zahalak (1982).

P. Morasso et al. / Trajectory formation

and handwriting

polygon) results from the partial time overlap of consecutive The present paper is mainly focused on the stroke-based trajectory formation and on its implications. Some empirical ing data are also presented for validating the model and for the stroke parameters.

85

strokes. model of handwritextracting

2. Strokes and trajectories: a generation mechanism Human manual movements are usually smooth, i.e. the trajectories traced by the hands have a continuous time course (between consecutive stop points) and do not show cusp points. In contrast, a polygonal curve, made up by a chain of straight or curved segments, is obviously discontinuous. However, there are different ways to deal with a polygon in the context of trajectory formation. For example, we can generate one side at a time (start stop start stop etc.): this discontinuous process will produce a discontinuous trajectory. Alternatively, we can generate two or more sides of the polygonal curve at the same time and, if they are appropriately timed and if we add up their contributions, such a discontinuous process will generate a continuous path, as shown in fig. 1. If we call “strokes” the sides of the polygon, it follows that in the latter case the strokes are not immediately detectable from the actual

Fig. 1. Examples of performance of the stroke composition model. The top-left part of each frame represents the simulated trajectory, uniformly sampled along the time axis. The underlying strokes are shown in the top-right part. For each frame, the lower trace is the tangential velocity profile, the higher trace is the curvature profile (vertical and horizontal calibration: arbitrary units).

86

P. Mormso

et (11./ Trajectory formation

und handwriting

movement pattern, but are “hidden”. In a previous paper (Morass0 and Mussa Ivaldi 1982) we discussed the mathematical properties of a mechanism of this kind and its relationship with the theory of spline functions [2]. Let us summarize here only the main features of the model: (i)

strokes are curved segments of given length, tilt angle and angular change (the geometric parameters) which are generated with a symmetric bell-shaped velocity profile centered at a given instant of time (the timing parameters) [3]; (ii) the timing parameters are chosen in such a way that for any time instant not more than two strokes are active; (iii) trajectories are generated by the linear superposition of the two currently active strokes. The generated trajectories result tangent to the polygonal curve of the underlying strokes at the midpoint of each stroke. In qualitative terms, the generation mechanism has a discontinuous input (the list of stroke descriptors) and a continuous output (the intended trajectory). Functionally speaking, this mechanism is placed at the interface between the motor planning level and the motor control level, which has the final output task of transforming an intended trajectory into the actual [2] The theory of spline functions (de Boor 1978) refers to interpolation by means of piecewise polynomial functions. Applied to the design of planar or spatial curves (by means of parametric equations) this theory allows to address the problem of shaping a curve by means of a chain of patches while assuring smoothness at the points of connection. In contrast with other interpolation techniques, in the case of spline functions the smoothness is not obtained by explicitly shaping each individual path (i.e. by solving simultaneous interpolation and continuity equations) but by dealing with shape and smoothness separately: in a first phase, the desired shape is approximated by means of a polygon and then, in a second phase, the vectorial contributions of the sides of the polygon are generated and superimposed, with activation delays chosen in such a way that for each time instant the same number of contributions is active. [3] The angular change is the angle between the initial and the final direction of a stroke. This parameter, similarly to the radius of curvature. measures the “curvedness” of a stroke but. differently from that, it is invariant to variations of size. The specific shape of the curved segments used to define the strokes is not really very important; we used circular arcs just because they need only one parameter to identify curvedness. Other parametric curved shapes could be used as well within the same computational approach, but they are likely to affect the global kinematic structure only beyond the level of curvature. The model of arm trajectory formation is a generalization of the method of spline functions for two reasons: (i) the model admits that strokes can be curved (spline functions are defined only for straight segments), (ii) the model requires only that the velocity profile of strokes are smooth bellshaped functions of time (spline functions deal only with polynomial segments).

P. Morass0 et al. / Trajectory formation

87

and handwriting

motor commands. Simple trajectories are generated by short lists of strokes and complex trajectories require structured lists of strokes [4]. In particular, handwriting requires a quite sophisticated symbolic structure, even if linguistic aspects are left out of consideration. The main point here is that the proposed trajectory formation mechanism provides a natural “alphabet” of strokes for expressing and generating handwritten traces. To give just an example, let us consider the computer generated script of fig. 2. The computer simulation program uses the trajectory formation mechanism previously discussed and, at a higher level, the program fetches the stroke parameters from a dictionary of the strokes of all characters of the alphabet, adding connection strokes when needed. As regards the dynamic aspects of trajectory formation, it is significant to look at the time course of the magnitude of the velocity vector. If the strokes are generated without time overlap (i.e. if the trajectory coincides with the polygonal curve) the tangential velocity profile is a sequence of completely separated bell shaped peaks. On the contrary, for an increasing degree of overlap, the peaks tend to blur and several

Fig. 2. Text generated

by a computer

program

which implemenl :s the stroke

c:omposition

[4] By “structured” we mean that complex patterns of movement can be built in a recursive lists of lists, similarly to the symbolic approach of artificial intelligence.

model. way as

different patterns may result. For example, if a big stroke is followed by a small stroke, only one velocity peak can be detected, but the second stroke induces an inflection point in the falling edge of the velocity curve of the first stroke. Furthermore, if two consecutive strokes are “nearly parallel” (i.e. the angle between the final tangent of the previous stroke and the initial tangent of the following one is small) the resulting velocity profile tends to be flat at the top. Finally, it is important to note that the proposed mechanism of trajectory formation intrinsically determines a fixed relationship between shape and speed: the points at which each stroke reaches its peak velocity have to be points of minimum curvature (the curvature of the corresponding stroke) whereas the minimum velocity points (i.e. those at which a waning stroke is taken over by a rising stroke) are bound to be peak curvature points.

3. Spatio-temporal

features

of

experimental

trajectories:

the

natural

speed

The experimental part of this work has been carried out on six subjects (four males and two females, aged from 24 to 40). The subjects were requested to produce pointing, handwriting and scribbling movements holding a stylus on a Summagraphics digitizing tablet. The latter was connected by serial line to a PDP 11/24 minicomputer and the system allowed sampling of x-y planar coordinates at a sampling rate of 50 points/s. In the following, the results concerning two subjects are presented; in qualitative terms, they are representative of the whole sample. Trajectories performed by human subjects are smooth at least up to the second time derivative and show a characteristic correlation between the velocity profile and the curvature profile (see fig. 3) which has been observed by several authors (Teulings and Thomassen 1979; Viviani and Terzuolo 1980; Morass0 and Mussa Ivaldi 1981; Abend et al. 1982). The segmented nature of these profiles suggests a discreteness of the generation mechnism which is the first lead to a stroke based model. With regard to the coupling between shape and speed, it may be asked how strong and how stable it is. Viviani and Terzuolo (1980) found that the tangential velocity profile of a handwritten letter or word keeps its

Fig. 3. Geometric-timing characteristics of hand trajectories (subject A). For each frame, the top part shows the recorded trajectory (sampling rate: 50 samples/s), the lower trace shows the tangential velocity profile (calibration: 500 mm/s), and the higher trace shows the profile of curvature (calibration: 0.008 l/mm). The same time calibration applies to all the frames; the duration of the leftmost displayed trace is 2 s.

d

z $. 5.

90

P. Morasso et al. / Trajectory formation

und hundwritlng

Fig. 4 (/e/c). Velocity profiles of straight hand trajectories performed with different speeds (subject B). The left part shows the recorded hand trajectories (sampling rate: 50 samples/set). The right part shows the corresponding velocity profiles (the sharpest profile corresponds to the leftmost trajectory, and so on). Velocity calibration: 500 mm/s. Time calibration: the duration of the displayed velocity traces is 5 s. Fig. 5 (right). Relation between duration and peak velocity of strokes, computed from slow and fast scribbles. The top and middle frames show two examples of fast and slow scribbles. executed by subject A. The upper part of both frames shows the recorded trajectory (sampling rate: 50 samples/s). The two traces underneath show: (V) the tangential velocity (calibration: 500 mm/s). (VP) its time derivative (no calibration is provided because only the zero crossings were of interest). The duration of the displayed traces is 10 s. The bottom frame shows the scatter diagram of the estimated stroke duration (T) vs the stroke peak velocity (V). The diagram stores data from 5 slow and 5 fast scribbles, for a total of 270 points. The displayed horizontal axis ranges from 100 mm/s to 1000 mm/s. The displayed vertical axis ranges from 440 ms to 1200 ms.

P. Mormso et al. / Trujecfoty formation and handwriting

91

structure when the writing movement is sped up from the natural speed (homotetic behaviour). However, if we slow down a movement, the homotetic behaviour will eventually break down, as it is shown in fig. 4 for the simple case of pointing movements. In this case, the movement performed at natural speed has a single peaked velocity profile, whereas for slower and slower movements more and more peaks show up. A behaviour of this kind suggests that the stroke generators have tunable timing parameters, but that the range of tuning is limited. This would be expected since it seems in general suspicious for a biological mechanism of motor control, unless it is in fact made up of several levels, to show a very large range of linearity. Furthermore, the observed behaviour is compatible with the following mechanism of speed control for planned trajectories: (i)

for a central range of speeds, the same geometric representation (i.e. sequence of strokes) is used, simply tuning the timing (durations and delays of the strokes); (ii) for lower speeds, the geometric representation is transformed into an equivalent and more redundant one (some strokes may be substituted by chains of smaller strokes, which give the same vectorial resultant); (iii) for higher speeds, the geometric representation can only collapse into a simpler one, by substituting some group of adjacents strokes into a vectorially equivalent one. According to this mechanism, the average stroke rate (number of strokes per second) should be independent of the overall speed of a movement. We investigated this hypothesis by recording scribbles (which have a large range of speeds) and by also asking the subjects to change the overall speed (to further increase the range of speeds). An example of slow and fast scribbles is shown in fig. 5. In order to estimate the degree of correlation between speed and stroke rate, the following computational procedure was followed: (i)

the time instants of the peaks of tangential velocity were identified (they can be better located looking at the negative going zero crossings of the time derivative of the velocity profile); (ii) each velocity peak (current peak) was associated with a stroke; (iii) stroke duration was estimated as the sum of the backward time

92

(iv>

interval (the difference between the current and the previous peak velocity time) and the forward time interval (the difference between the next and the current peak velocity time); a scatter diagram was built, with the stroke peak velocity on the abscissa and the estimated stroke duration on the ordinate and, finally, the correlation coefficient was computed.

Fig. 5 shows a scatter diagram which stores the data from five slow and five fast scribbles, each ten seconds long; the slow scribbles contribute particularly to the part of the diagram closer to the origin and the fast scribbles to the part farther from the origin. The correlation coefficient of the diagram passes the test of hypothesis that the population correlation coefficient is zero (n = 270, 0.95 significance level) and this is consistent with the hypothesis that stroke velocity and stroke duration are independent. Furthermore, if we consider the range of values of stroke duration, we may conclude that the stroke rate approximately ranges between two and four strokes per second. Finally, along the same line of reasoning, it is possible to say something about the natural speed or natural duration of manual movements: since a given band limited trajectory can be reproduced by a finite minimum number of strokes [5], the natural duration of such a trajectory can be estimated as the ratio between the required number of strokes and the mean stroke rate.

4. Fitting the generation mechanism to the experimental data The next step in validating the proposed model of trajectory formation consists of fitting it to the empirical data and of extracting the stroke parameters. From the computational point of view, this means to estimate the number of strokes and the geometric and timing parameters of each of them. The goodness-of-fit can then be estimated by simulating the model and comparing, at the same time, the generated trajectory and the corresponding velocity profile with the empirical data. [5] This conclusion can be derived by applying the sampling theorem of Nyquist or of Logan to trajectories instead of signals. It is known, indeed, that a finite number of samples is sufficient to reproduce a band limited signal and the classical theories provide also the corresponding interpolation techniques.

P. Morasso et al. / Trajectory /ormation

and handwriting

93

The fitting algorithm is based on some heuristic rules. One rule says that, since strokes have bell-shaped velocity profiles, each of them will determine a peak or an inflection point in the tangential velocity profile. The time instant of peak velocity of each stroke can then be directly detected from the velocity profile, whereas the initial and the final times cannot. With regard to this problem, we postulated (as a second rule, but only as a first approximation) that the initial time of each stroke coincides with the peak velocity time of the previous stroke and that the final time of the same stroke coincides with the peak velocity time of the following stroke. As a consequence, only two strokes are supposed to be active for any time instant, except for the peak velocity time of each stroke: at that time, the trajectory is fully determined by the only active stroke and it shares with it direction and curvature (this fact is true, a fortiori, also if the stroke duration is smaller than the one previously stated). Now, if we assume that strokes are circular, we can tentatively draw them as circles tangent to the previously identified points of the trajectory and with the corresponding radius of curvature: what we do not know yet is the angular change, i.e. the angle subtended by each stroke. This can be found, considering the consistency of the stroke representation of the trajectory, by computing the intersection between the subsequent circular arcs, already tentatively drawn. The resulting strokes are split into two parts, with respect to the point tangent to the trajectory: a forward semi-stroke and a backward semi-stroke. We can then define a coefficient of symmetry of a stroke as the ratio between the difference and the sum of the corresponding arcs (such coefficient ranges between - 1 and + 1). The computer program developed according to the algorithm is strongly interactive and proceeds through the following steps: (i) digitizing the trajectory, (ii) computing the speed and curvature profiles, (iii) searching the time instants at which the velocity profile shows peaks or inflection points, (iv) drawing circular segments tangent to the trajectory at the points which correspond to the previously identified time instants and with a radius of curvature given by the curvature profile, (v) computing the intersections among the circular segments. The

computations

above

give enough

information

to estimate

the

94

P. Morasso et al. / Trajec~otyvformation

Fig. 6. Fitting the model to empirical data. The simulated trajectory (M), and the corresponding The bottom frame shows the tangential velocity (M) (calibration: 500 mm/s). Time calibration:

and handwrrting

top frame shows the recorded trajectory (D). the chain of stroke (S) (sampling rate: 50 samples/s). profiles of the recorded data (D) and of the model the duration of the displayed velocity traces is 8 s.

geometric and timing parameters of the strokes, in particular the duration, the length, the angular change, and the coefficient of symmetry. A further necessary step, before simulating the model, requires to estimate the velocity profile of each stroke. A basic hypothesis of the model is that this profile is smooth and bell-shaped. Furthermore, the parameter estimation algorithm gives the value of the time integral of the profile, i.e. the length of the stroke by definition. From these constraints (shape, continuity, peak value, subtended area) and by means of some interpolation technique (we used spline functions, but it is not essential) the velocity profile of each stroke can be estimated. The model was then simulated and it yielded patterns similar to the recorded ones, but the simulated trajectories were systematically more

P. Morasso et al. / Trajectory formation

and handwriting

95

smoothed than the recorded ones in the sections of peaking curvature. We attributed this error to the initial hypothesis about stroke duration (or stroke overlap) and, in fact, by empirically adjusting the stroke overlap factor, we found that a value of 0.8 (i.e. a systematic 20% reduction of the initial estimate of stroke duration) yielded a good fit in many experimental situations, including the movements analysed in the following section. Fig. 6 shows an example of this process; the fit is particularly good, in this case, because the change of direction between subsequent strokes is quite sharp. 5. Histograms of stroke parameters The computational procedure described in the previous section allowed us to extract stroke parameters during handwriting. In particular, we compared the performance of two subjects (A: male, age 38; B: female, age 36) who were requested to write twenty words. Each word was four letters long and the same word was repeated twice (therefore, only ten different words were used). The recorded trajectories, in their sequence, are shown in fig. 7. The subjects started writing each word after a verbal command and with their natural speed (they did not know the sequence in advance). We examined the repeated words to check for artifacts and then we extracted the stroke parameters of the second execution of each word. The histograms of these parameters are shown in fig. 7, together with the corresponding means and standard deviations. It can be observed that strokes are symmetric, on average, that the average angular change is about 60 degrees, and that the average stroke duration is close to 700 ms. The differences between the average values of the parameters for the two subjects are not statistically significant. A significant difference between the two subjects was found instead in the average number of strokes per word: 13.6 for subject A and 15.6 for subject B (subject A produced a total of 273 strokes for the 20 words and subject B a total of 312). This experiment is simply a preliminary attempt in the analysis of handwriting, its main purpose being to link handwriting to other trajectory formation paradigms. More experiments are needed to fully characterize individual performance in handwriting and to discriminate inter-individual differences: two problems, these, in which metric and topological aspects are strongly interrelated (Bernstein 1957).

Fig. 7. Histograms of stroke parameters. Left-hand column: subject A; right-hand B. The top row shows the recorded words. The other rows show the histograms parameters of the extracted strokes: (i) duration, (ii) length, (iii) angular change,

column: subject of the estimated (iv) symmetry.

6. Discussion The main mechanism

purpose of this paper is to show how a discontinuous can produce continuous patterns of manual movements

P. Morasso et al. / Trajectory formation

without relying on the filtering action of motor system, which is probably not fit certainly quite busy in carrying out other control of motor impedance. The proposed mechanism (the stroke formation) refers to two different levels:

97

and handwriting

the mechanical part of the for the purpose while it is essential tasks such as the based

model

of trajectory

(i)

a level of symbolic representation of planned or partially planned trajectories which deals with lists (or lists or lists) of stroke descriptors, characterized by geometric and timing parameters; (ii) a level of process activation and process synchronization, where each process corresponds to an active stroke.

Strokes initiate, evolve, and die steadily during manual motor performance and a significant characterization of the mechanism is that a complete determination of the trajectory is not necessary before execution, since for each time instant the trajectory is under the influence of the two currently active strokes only. As a consequence, the process of spatial planning which feeds stroke parameters in a short term motor memory can evolve asynchronously with respect to the stroke generators and, in particular, it can back-track in case of emergencies or actions contingent to environment conditions. Strokes are still far from the muscles, they are abstract representations and processes which may allow the central nervous system to express and shape in a uniform way desired manual movements. Transforming these processes, which refer to the Euclidean space, into synergic arrays of motor commands, directly usable by the motor apparatus, is an additional level of motor control, whose consideration is however outside the scope of the paper. A final comment on the notion of stroke: why, after all, is it more convenient to think of movement units as hidden, abstract entities than as immediately detectable entities ? The reason, in our opinion, is essentially the same for which it is more convenient to synthesize sound or speech in the frequency domain than in the time domain. In the former case, the separation of the signal into elements is natural and the smooth blend between them during synthesis is implicitly (analogically) performed by the system “anatomy”. In the latter case, the connected nature of the acoustic signal causes a very great variability of the signal patches and joining them together is hard because it requires to explicitly examine segment to segment interaction.

98

References Abend. W., E. Bizzi and P. Morasso, 1982. Human arm traJectory formation. Brain 105. 331-348. Agarwal, G.C. and C.L. Gottlieb. 1977. Compliance of the human ankle joint. Transactions American Society of Mechanical Engineers, Journal of Biomechanical Engineering 99. 166-170. Benati, M., S. Gaglio. P. Morasso. V. Tagliasco and R. Zaccaria, 1980. Anthropomorphic robotics. I: Representing mechanical complexity. Biological Cybernetics 38. 125- 140. Bernstein, N., 1957. The coordination and regulation of movements. London: Pergamon Press. Boor, C. de, 1978. A practical guide to splines. Heidelberg: Springer Verlag. Cannon, SC. and G.I. Zahalak, 1982. The mechanical behaviour of active human skeletal muscles in small oscillations. Journal of Biomechanics 15, 111-121. Crane, H.D. and R.E. Savoie, 1977. An on-line data entry system for handprinted characters. Computer (March), 43-50. Herbst, N.M. and C.N. Liu, 1977. Automatic signature verification based on accelerometric data. I.B.M. Journal of Research Development (May), 245-253. Lashley. K.S.. 1951. ‘The problem of serial order in behavior’. In: L.A. Jeffress (ed.). Cerebral mechanisms in behavior. New York: Hafner. pp. 112-136. Mermelstein. P. and M. Eden, 1964. Experiments on computer recognition of connected handwritten words. Information and Control 7, 2255270. Morasso. P., 1981. Spatial control of arm movements. Experimental Brain Research 42, 2233227. Morasso, P. and F.A. Mussa Ivaldi. 1981. Un modele de generation des mouvements d’ecriture. Marseille: Journ& Thtmatiques. Forum Espace III. Position et Mouvement. Morasso, P. and F.A. Mussa Ivaldi. 1982. TraJectory formation and handwriting: a computational model, Biological Cybernetics 45, 131-142. 1979. Computer aided analysis of handwriting Teulings, H.L.H.M. and A.J.W.M. Thomassen. movements. Visible Language 13(3), 2188231. Viviani, P. and C.A. Terzuolo. 1980. ‘Space-time invariance in learned motor skills’. In: G.E. Stelmach and J. Requin (eds.). Tutorials in motor behavior. Amsterdam: North-Holland. pp. 525-533.