CONFIDENTIAL. Limited circulation. For review only.

Descending Commands to an Insect Leg Controller Network Cause Smooth Behavioral Transitions Brandon L. Rutter, Member, IEEE, Brian K. Taylor, John A. Bender, Marcus Blümel, William A. Lewinger, Member, IEEE, Roy E. Ritzmann and Roger D. Quinn, Member, IEEE

Abstract— Biological inspiration has long been pursued as a key to more efficient, agile and elegant control in robotics. It has been a successful strategy in the design and control of robots with both biologically abstracted and biomimetic designs. Behavioral studies have resulted in a good understanding of the mechanics of certain animals. However, without a better understanding of their nervous systems, the biologically-inspired observation-based approach was limited. The findings of Hess and Büschges, and Ekeberg et al. describing the neural mechanisms of stick insect intra-leg joint coordination have made it possible to control models of insect legs with a network of neural pathways they found in the animal’s thoracic ganglia. Our work with this model, further informed by cockroach neurobiological studies performed in the Ritzmann lab, has led to LegConNet (Leg Controller Network). In this paper we show that LegConNet controls the forward stepping motion of a robotic leg. With hypothesized additional pathways, some later confirmed by neurobiology, it can smoothly transition the leg from forward stepping to turning movements. We hypothesize that commands descending from a higher center in the nervous system inhibit or excite appropriate local neural pathways and change thresholds, which, in turn, create a cascade of reflexes resulting in behavioral transitions.

I. INTRODUCTION

B

iological systems have been a source of inspiration for robotic control at varying levels of abstraction and complexity. In the design of legged machines this has ranged from the highly abstracted Whegs™ [1], through less abstract but still highly simplified systems like RHex [2] and MechaRoach [3, 4], to more flexible and complex systems, including Robot II [5], the TUM walking machine [7] the Tarry series [8, 9], the Lauron series [10], and BILL-Ant-p [11].

Manuscript received March 28, 2011. Supported by the U.S. AFOSR under grant FA9550-07-1-0149 B. L. Rutter, B. K. Taylor and R. D. Quinn are with the Mechanical and Aerospace Engineering Department, Case Western Reserve University, Cleveland, OH 44106 USA. (216-368-5216; e-mail: [email protected]; [email protected]; [email protected]) J. A. Bender and R. E. Ritzmann are with the Department of Biology, Case Western Reserve University ([email protected]; [email protected]) M. Blümel is with the Department of Animal Physiology, Institute for Zoology, University of Cologne, 5023 Cologne, Germany ([email protected]) W. A. Lewinger was with the Department of Electrical Engineering and Computer Science, Case Western Reserve University. He is now with the Institute of Perception, Action and Behavior, University of Edinburgh, Edinburgh, UK ([email protected])

As engineers wish to know more about the details of animal neuromechanics, modeling has become more important as a tool to aid biologists in their research. Robot models of animal systems can be used to test biological hypotheses. A hardware model has the advantage that its complete state can be measured at all times during an experiment, which is impractical in an animal. The engineer of legged systems and the biological modeler have different motivations. The primary question for the engineer is: how do we best make a machine move? We desire agile control of movement with low computational requirements. For the biological modeler, the primary question is: how do animals move, and how do we make something that moves sufficiently like an animal to serve as a model? Though different, these desires are complementary, and can be simultaneously addressed in the same effort. Robotic models of legged animals have been mostly behaviorally based. As described in [12], the behavioral approach has drawbacks in terms of usefulness to biology – examining a black box does not generally help to understand the mechanisms that produce the behavior. Understanding these biological mechanisms also unlocks the principles of design and control that engineers seek. Inspiration from biology can be an effective way to gain agility and efficiency. In this process, neural networks, dynamical central pattern generators, and finite state machine models of control structure or behavior can be used as tools of implementation. However, the design of effective controllers without detailed knowledge of animal neuromechanics is a difficult problem; see discussion in [13, 14]. Developments during the last decade in our understanding of insect walking systems, encapsulated in the study of neural mechanisms of stick insect leg coordination by [15, 16], have made it possible to construct and control robotic model legs based on known neural and mechanical properties of biological systems. At a higher level of abstraction is the review of coordination of “multisegmental organs” [17], including legs and spinal cords. Büschges presents evidence that pattern generators often are distributed for control of physical segments, and mentions that we need to know the internal organization so that we can target and analyze the neural mechanisms for taskspecific flexibility. Such biologically-based models can provide appropriate

Preprint submitted to 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. Received March 28, 2011.

CONFIDENTIAL. Limited circulation. For review only. paths to investigate and predict the effects of descending commands in the transition between and generation of different behaviors in local control. Systems here and in [18] were constructed primarily for biological modeling of step generation in insects, particularly to address questions about the role of higher-level influences on transitions between widely varying behaviors in local leg control systems. The implementation of controllers capable of generating insect stepping behavior in the manner described by [16], as in [18, 19], led to the development of the Leg Controller Network (LegConNet) described in this paper. LegConNet is based upon neural connections discovered in the thoracic ganglia of stick insects, which contain the local neural circuits responsible for the coordination of joints in a leg. It has been expanded based upon hypothesized connections being researched in insects, some having since been confirmed. Previous results have shown that a robotic model of an insect leg controlled by LegConNet steps in a forward walking behavior and adapts to irregularities in the terrain [20]. It was also previously shown that a linear muscle model provided it with greater robustness such that it could step indefinitely despite disturbances [18]. A simplified version was implemented in the hexapod BillAnt-a, which was shown to walk over irregular terrain and use high-level descending commands to modify the walking behavior in order to create turning actions and seek out goals [21]. In this paper, LegConNet is shown to control a robotic model of a cockroach leg in behavioral transitions from forward walking to insect-like turning. II. BASIS IN STICK INSECT NEUROBIOLOGY The model controller from [16] was the basis for LegConNet which consists of independent joint control systems. In this system each joint controller is a bi-stable pattern generator, and the coordination task of each joint pattern generator is to determine whether to be in the joint’s flexion or extension state. To make this decision the joint controller has access to specific sensor data as in the animal, but it has no direct central information about the state of the other joint controllers. This reflects the effective lack of central coupling in the biological archetype. For forward stepping of an insect leg model, the appropriate pathways for each joint’s movement action can be derived almost completely from the literature; only the threshold values have to be adjusted to function with a particular leg geometry. The basis for LegConNet for forward stepping of the middle leg of a stick insect, along with the degrees of freedom (DOF) of the stick insect middle leg are shown in the sensory flow diagram of Figure 1 [16]. As an example, the blue module controls the action of the femur-tibia (FTi) joint. The FTi joint has a single degree of freedom, where one action flexes the joint (FLX) and another extends it (EXT). The dynamics of an action consist of neural/control behavior, plant mechanics and environmental interaction, and the relevant action output is the torque and/or movement

that results at that joint.

Forward

Figure 1 Sensory-flow diagram of the stick insect left middle leg for forward stepping (as described in [16]). The Degrees of Freedom (DOF) are as follows. ThC (Thoraco-Coxal) protraction and retraction, CTr (CoxaTrochanter) levation and depression, and FTi (Femur-Tibia) flexion and extension. The leg segments, from the body outward, are the coxa, femur and tibia.

Switching between actions of a joint is influenced by sensory signals, which are affected by actions at other joints. These inter-module influences couple the modules (joints) via sensory feedback making the system a set of Sensory Coupled Action Switching Modules (SCASM) [19]. The behavior of the entire system depends on the strength and sense of the inter-module connections, together with the control dynamics of each action, the physical dynamics of the system, including mechanical coupling between modules, and the environment. In Figure 1, the arrows between modules represent this sensory coupling. The FTi angle sensor signal, for instance, influences the switching between actions of both the FTi module and the CTr module. The sensory-flow representation above does not fully describe behavior. The sensory arrows represent the direct sensory influences between modules, but not which actions are promoted, or what level of sensory activity is necessary to do so. We can generate a more explicit representation of the sensory connections using an event-space diagram, shown in Figure 2.

Figure 2 Event-space diagram for forward stepping of the stick insect left middle leg. The double-line actions in each module specify system dynamics that usually lead to the double-line sensory events, and solid-line actions likewise lead to solid-line events. Color encodes the module most directly involved in generating each sensory event.

Transitions which may occur between actions are shown with arrows. Rather than explicit sensory signals, the influences on action switching are represented here as sensory events. An example from the FTi joint of the stick

Preprint submitted to 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. Received March 28, 2011.

CONFIDENTIAL. Limited circulation. For review only. insect leg would be “FTi joint is extended”, which encourages both CTr depression and, in combination with leg load, FTi flexion. The simplest example of sensory influences in the stick insect leg are the load influences on the ThC module. Leg load promotes retraction, meaning that if the leg is touching the ground, the ThC will rotate rearward, driving the animal forward. Leg unload encourages protraction, meaning that if the leg is unloaded, the ThC joint will rotate forward. Retraction while on the ground and protraction while in the air produce forward stepping. III. LEGCONNET PRODUCES THREE LEG BEHAVIORS

The model cockroach leg shown in Figure 5 was used in these experiments. It is a 10.1:1 scale 4-DOF model of the left middle leg of the cockroach Blaberus discoidalis. The segment lengths are coxa = 87.2mm, femur = 93.2mm, and tibia = 74.7mm. The coxa, femur and tibia segments are solid machined aluminum, while the connection between the motors for ThC1 and ThC2 are a plastic-aluminum sandwich beam to increase stiffness. A rounded, smooth plastic foot simulates the low-friction oiled-plate environment often used in animal experimentation [23]. An AI-series servomotor from Mega Robotics (Megarobotics Co., Ltd. Seoul, Korea) actuates each joint; the ThC1 motor is model 1001, the others are model 701.

Experiments in this study involve transitions between three distinct behaviors observed in cockroach middle legs [22]. The DOF of a cockroach middle leg, along with an illustration of the experiments performed are presented in Figures 3 and 4. In forward stepping (FWD) the FTi joint extends and the CTr joint depresses during stance. In the Inside Turn Forward (ITF) behavior, the FTi joint flexes during stance and extends during swing, opposite of FWD, but the rest of the CTr behavior is similar to forward walking. In the backward type of inside turn (ITB) the CTr joint levates during stance and extends during swing, opposite of FWD, but the FTi movement is similar to ITF.

Figure 5 Model cockroach left, middle leg on mount and ready for an experimental run. Photo taken from rear: The aluminum base element pointing down and to the right is parallel to the simulated body centerline.

Figure 3. Dorsal view of the right middle leg of a cockroach and its DOF. Thoraco-Coxal (ThC) angle 1 and angle 2 motion corresponds to leg promotion/remotion (i.e., lowering/lifting the leg) and adduction/abduction (i.e., moving the leg away from or closer to the body centerline) respectively. Coxa-Trochanter (CTr) angle motion corresponds to levation/depression (i.e., extended/flexed) of the CTr joint, and Femur-Tibia (FTi) angle motion corresponds to flexion/extension of the FTi joint. Rostral is tail to nose.

Figure 4. Three cockroach walking behaviors observed in middle legs. Forward walking (FWD). Inside turning in forward direction (ITF). Inside turning backwards (ITB). Rostral is tail to nose. Red dashed lines with arrows indicate direction of motion during stance. The beginning of stance is shown in gray and the end in black. The coxa motion is not shown.

The pathways that are active in LegConNet that implement each of these behaviors are shown in the eventspace diagrams in Figures 6, 7, and 8. The FWD pathways established for stick insects [16] were translated to this cockroach leg. Differences result from the differences in morphology of the two animals’ legs. The cockroach’s leg is sprawled (Figure 5), whereas the stick insect’s leg is more upright. This led to a reversal of the functions of the ThC and CTr joints in LegConNet as can be seen in a comparison of Figures 1 and 3. ThC and CTr control overall leg levation/depression in the cockroach and stick insect, respectively. The pathways for the ITF and ITB behaviors were hypothesized based upon incomplete data, but some have since been confirmed (Büschges personal communication). Figure 7 for ITF is also marked with numbers indicating parameters that must change from the FWD configuration in order to accomplish ITF, together comprising the configuration changeset ΔITF. It is hypothesized that, to change from forward walking to inside turning, commands descending from a higher center would contain this changeset information. The load influences on FTi actions and the FTi angle effect on ThC1 actions are reversed. The thresholds for the FTi angle on FTi action and CTr angle on ThC1 action pathways have also changed. Though not

Preprint submitted to 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. Received March 28, 2011.

CONFIDENTIAL. Limited circulation. For review only. explicitly related to this behavior, reversal and modification of local sensory influences via descending commands in B. discoidalis has been shown in [24]. The event-space diagram for ITB is shown in Figure 8, along with the parameters which must be changed to go from ITF to ITB: Δ ITB. These changes are a reversal of load/ground contact influence on CTr, and a reversal of CTr angle influence on ThC. Even though LegConNet is shown with a different event space diagram for each of the three different behaviors for clarity, it is actually one network consisting of all of these pathways. Descending commands excite or inhibit certain pathways and modify thresholds to encourage a particular behavior.

Figure 8. Event space diagram for ITB behavior. Switching events which have changed from ITF are marked with a yellow shadow; all changed parameters are noted in boxes attached to switching event lines or transition paths. The effect of load on CTr and the effect of CTr angle ThC have both been reversed from the ITF event space configuration.

IV. BEHAVIORAL TRANSITIONS

Figure 6. LegConNet event space diagram for cockroach forward stepping (FWD).

Experiments were run with LegConNet controlling the cockroach leg model to examine its behavior during transitions between behaviors. “Descending commands” consisting of the changesets ΔITF and ΔITB were applied. In the first experiment (Figure 9), the leg is in the FWD behavior when the changeset ΔITF is suddenly applied at 20s and transitions to the ITF behavior. ΔITB is then applied at 40s, leading to the ITB behavior. Figure 9 clearly shows changes in phase between joint angles in each case as expected. The transient behaviors at the switching points quickly decay to the appropriate steady state behavior. Three repetitions of this experiment were run, all producing similar results. See accompanying video.

Figure 7. Event space diagram for ITF behavior. Changed switching events are marked with a yellow shadow; changed parameters are noted in boxes attached to switching event lines or transition paths. The effect of load on FTi and the effect of FTi angle ThC have both been reversed from the FWD event space configuration.

Figure 9. Joint angles and state of the Thorax-Coxa1 DOF during behavioral transitions; remotion (i.e., when the leg is lifted from the ground) is highly correlated with swing. The leg is in the FWD behavior when the changeset ΔITF is applied at 20s and transitions to the ITF behavior (top). ΔITB is then applied at 40s, leading to the ITB behavior (bottom). See accompanying video.

Preprint submitted to 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. Received March 28, 2011.

CONFIDENTIAL. Limited circulation. For review only. (a) shown in Figure 12, inappropriate behavior is generated before the gradual adjustment of ΔITFthresh brings the system to ITF. One of these (a) experiments became stuck in the transient and required a perturbation to complete the transition.

Figure 12. Experiment (a): Transition rule change at 10s before threshold changes begin causes inappropriate, uncoordinated behavior; gradually changes to ITF with threshold changes. Figure 10. Foot path in x-y horizontal plane during the FWD to ITF transition in the range from 18s to 22s. Unlabeled arrows indicate path direction, labeled arrows point to the stable behaviors and the location of the foot at 20s when ΔITF is applied.

Figure 10 shows the path of the foot in the x-y horizontal plane during the FWD to ITF transition during the range from 18s to 22s. The small closed curve is the foot path for steady state forward stepping. ΔITF is applied at 20s at the point indicated in the path. The foot path immediately changes and starts to trace a curve typical of an ITF behavior. We conclude that transitions between behaviors can occur smoothly and during one step when the changesets are applied simultaneously. Figure 10 shows a transition from FWD to ITB when the combined changeset ΔITF + ΔITB is applied at 20s. This experiment was run three times with a successful transition in each case.

Figure 11. Joint angle data for forward to inside turn-backward at 20s. Note the change in the range of motion of the CTr angle, and the phase of ThC1with respect to the other joints, which happen within one step.

If descending commands do not arrive at once, can the leg transition between behaviors? Experiments were run to investigate the behavior of the leg during gradual transitions. Given the implementation of the control system, rules (pathways) can only be changed instantaneously. However, thresholds can be gradually transitioned. Three types of experiments were run multiple times. In each case the thresholds were linearly changed to those for ITF starting with the leg in the FWD behavior. In the three experiments, rules were changed (a) before, (b) in the middle of, and (c) after the threshold changes. In typical results for experiment

Experiment (b), shown in Figure 13, needed no perturbations and generated relatively smooth transitions. For experiment (c), shown in Figure 14, in all three repetitions, the oscillatory behavior gradually died out at about 15s and eventually stopped, but was restored when ΔITFtrans was applied at 25s.

Figure 13. Experiment (b): Transition rule change at 15s in middle of linear threshold change starting at 10s. Successful transition with a spastic-looking phase near the transition rule change.

Figure 14. Experiment (c): Oscillatory behavior stops after transition thresholds have changed beyond a certain point (about halfway to complete change). Transition rule change at t=25s brings system to ITF.

V. CONCLUSIONS LegConNet is based upon a network discovered in a thoracic ganglion of stick insects that coordinates the joints of a leg in adaptive forward stepping. LegConNet was modified for a robotic cockroach leg based upon the different functions of its joints. It was shown to produce reliable forward stepping, and was then modified to also produce two types of turning behaviors observed in the cockroach. Descending commands to the local system in the thoracic

Preprint submitted to 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. Received March 28, 2011.

CONFIDENTIAL. Limited circulation. For review only. ganglia from a higher center in the cockroach nervous system trigger different behaviors manifested in movements of its legs. Experiments were run with LegConNet to determine if parameter changesets (descending commands) could cause smooth behavioral transitions in the model cockroach leg. Transitions between behaviors occur smoothly during system operation when the necessary parameters are changed all at once. Also, changesets that are subsets of another changeset can be added to reach the final behavior. Applying changesets gradually over a period of time is more problematic. The time over which a changeset is applied has an effect on system function during that time, and the coordination of the changes applied can have a strong effect on system output. Sensory event thresholds may have a narrow range of acceptable coordination with the transition rule changes, which reflects the possibility that both of these types of parameters may be associated with the same neural excitation parameters in an animal. The best results were achieved when the transition rules were changed in the middle of the sensory threshold changes (Figure 13). Therefore, we believe that the sense of the inequality in the transition rule should be changed when that sensory threshold passes through zero. When the control system is implemented as a true network with no discrete rules, the current transition rules can be also gradually changed with their connection weights being ramped up and down. Also, the duration of the descending commands arrival and the transitions they cause are much more rapid than the experiments performed here, probably on the scale of tenths of a second or less. Therefore, the experiments with simultaneous changesets are closer to reality. These ideas will be tested in future experiments.

[8] [9] [10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

[20]

REFERENCES [1]

[2]

[3]

[4]

[5]

[7]

T. J. Allen, R. D. Quinn, R. J. Bachmann, and R. E. Ritzmann, "Abstracted Biological Principles Applied with Reduced Actuation Improve Mobility of Legged Vehicles," in Intelligent robots and systems; 2003 IEEE/RSJ international conference on intelligent robots and systems (IROS 2003), Las Vegas, NV, 2003, pp. 1370-1375. R. Altendorfer, N. Moore, H. Komsuolu, M. Buehler, H. B. Brown, D. McMordie, U. Saranli, R. Full, and D. E. Koditschek, "RHex: A biologically inspired hexapod runner," Autonomous Robots, vol. 11, pp. 207-213, Nov 2001. M. Boggess, R. Schroer, R. Quinn, and R. Ritzmann, "Mechanized Cockroach Footpaths Enable Cockroach-like Mobility," in International conference on robotics and automation; 2004 IEEE, New Orleans, La., 2004, pp. 28712876. T. E. Wei, R. D. Quinn, and R. E. Ritzmann, "A CLAWAR That Benefits From Abstracted Cockroach Locomotion Principles," in Climbing and walking robots Conference, Madrid, Spain, 2004, pp. 849-858. K. S. Espenschied, R. D. Quinn, R. D. Beer, and H. J. Chiel, "Biologically based distributed control and local reflexes improve rough terrain locomotion in a hexapod robot," Robotics and autonomous systems, vol. 18, p. 59 (6 pages), 1996. F. Pfeiffer, H. J. Weidemann, and J. Eltze, "The TUM Walking Machine. - In: Intelligent Automation and Soft Computing," in Trends in Research, Development and Applications. vol. 2: TSI Press, 1994, pp. 167-174.

[21]

[22]

[23]

[24]

A. Buschmann, "Home of Tarry I & II: design of the walking machine Tarry II," 2000. A. Buschmann, "Home of Tarry I & II: frequently asked questions about Tarry," 2000. B. Gassmann, K.-U. Scholl, and K. Berns, "Behavior control of LAURON III for walking in unstructured terrain," in Intl. Conference on Climbing and Walking Robots (CLAWAR '01), Karlsruhe, Germany, 2001, pp. 651-658. W. A. Lewinger, M. S. Branicky, and R. D. Quinn, "Insectinspired, actively compliant robotic hexapod," in International Conference on Climbing and Walking Robots (CLAWAR), London, U.K., 2005. B. Webb, "Can robots make good models of biological behaviour?," Behavioral and Brain Sciences, vol. 24, pp. 10331050, 2001. A. J. Ijspeert, A. Crespi, D. Ryczko, and J. M. Cabelguen, "From swimming to walking with a salamander robot driven by a spinal cord model," Science, vol. 315, pp. 1416-20, Mar 9 2007. A. J. Ijspeert, "Central pattern generators for locomotion control in animals and robots: a review," Neural Netw, vol. 21, pp. 64253, May 2008. D. Hess and A. Büschges, “Sensorimotor Pathways Involved in Interjoint Reflex Action of an Insect Leg,” in Journal of Neurobiology 1997; 33(7):891-913. Ö. Ekeberg, M. Blümel, and A. Büschges, "Dynamic simulation of insect walking," Arthropod structure & development, vol. 33, pp. 287-300, 2004. A. Buschges, "Sensory Control and Organization of Neural Networks Mediating Coordination of Multisegmental Organs for Locomotion," Journal of neurophysiology, vol. 93, pp. 11271135, 2005. B. L. Rutter, W. A. Lewinger, M. Blümel, A. Büschges, and R. D. Quinn, "Simple Muscle Models Regularize Motion in a Robotic Leg with Neurally-Based Step Generation," in ICRA 2007, Rome, 2007. W. A. Lewinger, B. L. Rutter, M. Blümel, A. Büschges, and R. D. Quinn, "Sensory Coupled Action Switching Modules (SCASM) generate robust, adaptive stepping in legged robots," in CLAWAR 2006: 9th International Conference on Climbing and Walking Robots Brussels, Belgium, 2006. B. L. Rutter, W. A. Lewinger, B. K. Taylor, M. Wilson, M. Blümel, Ö. Ekeberg, A. Büschges, R. E. Ritzmann, and R. D. Quinn, "Neurally-based robot control for neuromechanical modeling of insect stepping. Program No. 449.13. ," in 2006 Neuroscience Meeting Planner Atlanta, GA: Society for Neuroscience, 2006. W. A. Lewinger and R. D. Quinn, " A Hexapod Walks Over Irregular Terrain Using a Controller Adapted from an Insect’s Nervous System," in International Conference on Intelligent Robots and Systems (IROS) 2010, Taipei, Taiwan, October 1822, 2010 Mu, L. and R. Ritzmann (2005). "Kinematics and motor activity during tethered walking and turning in the cockroach, Blaberus discoidalis." Journal of Comparative Physiology A: Neuroethology, Sensory, Neural, and Behavioral Physiology 191(11): 1037-1054. A. K. Tryba and R. E. Ritzmann, Multi-joint coordination during walking and foothold searching in the Blaberus Cockroach. I. Kinematics and electromyograms. J Neurophysiol 83:3323– 3336, 2000. L. Mu and R. E. Ritzmann, “Interaction between descending input and thoracic reflexes for joint coordination in cockroach. II Comparative studies on tethered turning and searching,” in The Journal of Comparative Physiology A, 194:299 – 312, 2008.

Preprint submitted to 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. Received March 28, 2011.