DESIGNING AND PORTABILITY OF INDIAN CLASSICAL DANCE HAND GESTURES SANJAY GOEL, PRIYANK SINGH, et al Jaypee Institute of Information Technology, India ______________________________________________________________________________________ Indian classical dance is well known for its beauty and artistic gestures. In the classical dances hand gestures represent particular meanings like God, animal, feelings and other symbols. These gestures are not only used in dance but also used in Yagna and Vedas. Existing software systems do not provide any tool for designing these hand gestures. Image and graphics database are used by dance scholar. This dissertation proposes a potentially portable system for designing hand gestures. Interface is provided such that a dance scholar use can easily interact with it with little training. System is using Extensible 3D Graphics based language for playing and storing the animation which makes it portable for any platform. _______________________________________________________________________________________

1. INTRODUCTION The Vedic Hymns and Mantras or mystical syllables, are forming the backbone of Hindu holy worship. The recitation of the Hymns and Mantras are sometimes accompanied by hand gestures and movements. These hand gestures are known as Mudras or Hastas. Mudras are the yoga of the fingers. However the origins of the Mudras are not sure but they are used in various cultural and artistic activities like in religion dance and in art. Advancement in technology and use of computer can increase the usability and learning of Mudras. However there is no such software exists which helps in identifying these Hand Gestures and provides tutorial for Mudras used in dance and art. This dissertation proposes a system that models human hand gestures and discusses its components and applications. Deliverables of this system are user-friendly and inexpensive interface for designing hand gestures; platform for designing animation for numerous applications and scripts based output for increasing the portability of the animation and to reducing the size

2

•

S.GOEL, P. SINGH,

et al

of the 3D animation from MBs to few KBs. Developed system can be used for different applications like sign language, gaming and development of pictographic symbols.

2. LITERATURE SURVEY Theme of the project wonders around Indian Classical dance and designing of hand gesture. Before starting the project a detailed overview of Mudras- hand gestures used in Indian Classical Dance and other artistic and cultural should be known. Apart from it, existing hand gesture based coding systems and various fields where we can apply these gestures should be known because it will help to increase the applicability of the product in different areas. Lastly, current trends of using scripting language for 3D animation and designing are discussed. Even some of the systems which are developed specifically for the animation programming are reviewed. Literary survey can be divided in parts where first part focuses on Indian Classical Dance and second part focuses on existing system which provide such functionality.

Indian classical dance is the combination of different Mudras and Vedic Mantras. Origin of these Mudras is Veda and Yagna for the old time specific hand gestures are used to define different meaning sometimes it represents God some time it represents element or some time animals. Hand gestures are widely used in different Hindu rituals. Mudra is basically originated from “Mud” meaning “Bliss”, this bliss is attained through the divine power of hand gestures or Mudras. After doing a brief survey mainly two categories are found. Which are “Asamyukta hastah” (hands, not united) [22, 23] these are single hand gestures which can be performed by either using left hand only either by using right hand only or by both hands without combining two hands. Second category refers to those Mudras which require both hands simultaneously for performing gesture which is known as “Samyukta Hastha”( united or combined and) [22, 23]. According to the ancient scripture “Abhinaya Darpana” there are 28 single hand gestures and 24 united hand gestures. All these different hand gestures or Mudras are frequently used in Indian Classical Dance forms especially in Bharatnatyam. These hand gestures define God, relationship, different castes also nine planets rivers animals and so many things. Each these Mudras have Sanskrit names and every time Mudras are thought person who

Technical Report, Department of Computer Science, JIIT

•

3

is learning should recite of the dance. Mudras are like audio visual component for representing particular meaning. Some Single handed and double handed Mudras are mentioned below [24].

Mayur Used to express peacock, creeper, bird of omen, throwing up, stoking, hair the flow and direction of river flow etc

SinhaMukh Used to express animals like rabbit elephant, lion face, grass, lotus petals, preparing, medicine, garland of lotus

PadmaKosh Used to express fruit, ball, flower bud to eat, mango, bunch of flowers, lotus, anthill, breast, eating meals, rain, and bell shapes

Asamyukta Mudras - Single Hand Gesture The currently available systems for human computer interaction that utilize human hand gestures as a source of input have been divided into three main categories – Vision based, Glove based and Robotic arm based. Each of these systems has certain inherent flaws which are impeding the growth and widespread application of this method of interfacing with personal computers. The limitations of current machine vision technology hinders the accuracy of existing vision based systems [13,14, and 16] which attempt to recognize human hand gestures by video and image processing techniques. Similarly, the cost of the glove [4, 5, and 6] based technique and the limited number of gestures that a robotic arm [15] can model; all prove to be major drawbacks for the commercial applicability of this system. This dissertation provides an inexpensive mechanism for utilizing electronics sensors placed on the human hand to detect coarse hand gestures. This

4

•

S.GOEL, P. SINGH,

et al

technique shall possess the correct mix of costing, accuracy and hand movements for it to be used as a base for other applications. Human hand modeling systems are been currently utilized in virtual reality, computer animation and gaming. Researchers at the Hong Kong University developed a naïve system for simulating hand gestures which was based on a static image of the final gesture as input [2]. A highly detailed and specialized tool for synthesizing hand gestures specifically for musical notes has been proposed in [3]. The authors have discussed in detail concepts for a smooth animation of human hands specially the motion of the joints between the fingers. In a series of publications [7, 8] researchers have presented a Handrix which is used to codify hand gestures by the use of a unique programming language. A separate section of the presented system [9, 10] also analyses human gesture images and attempts to replicate them using HACS. Some system are particularly designed for interacting with 3D objects but these systems use very expensive methods for taking input [4, 5, and 6]. For making user friendly some system are using voices for designing 3D objects [20]. Some interactive systems are using hand gestures for manipulating and designing 3D objects. For example, a context-based manipulation is used like extending, shorting and holding [21] of the objects but it has very limited number of operations. Apart from it, a very expensive and interactive system is designed in Oita University Japan [18], which helps in designing a 3D object with the help of the hand gesture. Similarly, two handed gesture interface is developed which is TGSH (two-handed gesture environment shell) [19], that helps in manipulating the objects. Hand gestures are also used for performing.

The modeling scheme presented in this proposal suggests the use of a scripting / markup language to codify hand gestures. With the influx of scripts and languages of the X-Family making forays into current web application standards [11] and graphical applications in upcoming operating systems such as Microsoft’s Windows Vistas [12], the development of a scripting language for the modeling of hand gestures can increase its applicability manifold. An example of such scripting / markup languages can also be found in the use of VRML (Virtual Reality Modeling Language) a markup language designed for representing 3D graphics specifically over the World Wide Web [17]. Some of the existing system use combination of markup language to develop 3D environment one such system is Virtual Human Markup Language [25], it’s

Technical Report, Department of Computer Science, JIIT

•

5

the combination of various scripting languages which separately deals with the speech human emotions human body animation and but application which this language target is 3D face designing including emotions. Some languages which use concept of Actor and role based animation control schema [26 and 27] is introduced which helps in designing of 3D virtual reality models with the help of scripting language but the format supported by this system is not universal so it’s big disadvantage for this system. Body Animation Markup Language [28] is developed for the purpose of animating human movements with the help of animation scripting language but the system doesn’t provide support for laymen users so its very complicated to use it.

3. INDIAN CLASSICAL DANCE LEARNING TOOL This tool provides user a platform for modeling hand gestures based animation. Unlike other existing tools it does not depend on static content. Output is in the form of the 3D animation and caption which makes it more interactive. Animations are stored in the form of the script files which provides the portability to system as well as due to small size it becomes easy to use over internet. The software is designed using the concepts of Object Oriented Programming. Each component is designed in the form of class so that it becomes easy to extend the project as well as to add the functionality in it. The whole software is developed in four phases. First phase was focused on modeling of the hand models and interface, second phase consist of designing mathematical transformations for modeling different hand gestures in the third phase these transformations are mapped on the script language to provide portability and in final phase all these components are integrated together.

It uses 3D hand models for animating the hand gestures. These hand models are developed through 3D rendering tools as we can see both wireframe as well as textured model in Fig.1. All mathematical transforms are mapped on these models.

6

•

S.GOEL, P. SINGH,

et al

Fig. 1(a) Textured Hand Model

Fig. 1(b) Wireframe hand model

Interface is designed in such way so that it becomes interactive for the user. Interface consists of four sections. These sections are Animation Control Bar, Animation Control Panel, Selection Control and Animation Arena. Hand Model

Animation Bar Selection Control

Animation Control

Fig. 2. Tool for modeling hand gestures 3.1. Animation Control Bar Animation is designed as a collection of frames. These frames are linearly mapped on the Animation Bar. User can easily manipulate animation by moving control bar to a particular frame position and then by adding the transformation. Predefined gestures can be used in animation only user has to load the script file of predefined gesture in new animation from particular position of the control bar.

Technical Report, Department of Computer Science, JIIT

•

7

3.2. Animation Control Panel Animation Control Panel controls the file loading and saving operations. Also basic operations like play pause and replay are handled by this particular component. It also includes one textbox for adding caption with the animation frame. 3.3. Selection Control Selection control panel used to select a particular component of the hand. It is the combination of the check boxes which is used to select the different part of the hand like finger, wrist and others. User can easily select multiple components and transform them into desired gesture with the help of the keys. 3.4. Animation Arena This is the main visual component of the software. It is the combination of caption and hand models. Caption is displayed on the top of the animation along with each frame. User applies transformations on the hand models for modeling gestures.

It is incumbent that the naïve user be able to easily create animations using this tool. Hence application was developed to enable seamless interfacing to the modeling part of the software. Figure 2 shows an in progress design of a gesture animation. The user can select one or more hand segments via the use of check boxes and can then use keystrokes to animate the hands akin to playing a video game. The animation has a universal track bar which allows the user to interlace multiple movements of multiple hand segments without having to do so simultaneously. The user can also save frequently used movements of individual segments into the segment database and can retrieve then instantly with the click of a button. Finally, the entire gesture animation can be saved as a gesture script and multiple gesture scripts can also be interlaced to form a longer animation Saved gesture scripts can be used to create longer connected animations. The hand gesture animator gives the user a stripped down version of the main application which allows the use to simply load and play the animation. Portability of a designed gesture was one of the chief concerns while developing this software. This portability has been achieved by an XML based gesture script. XML based scripts are the current trend for storing information about 3D graphics and hence the use of a gesture script makes porting

8

•

S.GOEL, P. SINGH,

et al

to other applications easier. The gesture script is made up of a set of tags. It contains a total of fourteen tags for each of the fourteen segments of both the arms. Within these tags time stamped information about the transformations has been stored. This makes parsing, retrieving, and populating the linear storage containers very fast and easy. This gesture script also contains textual keywords which the user wishes to display during the animation.

4. RESULT Application delivers a platform for developing different hand gestures based animation. Since targets are laymen users so interface is tried to keep as simple as possible. Some major delivers are discussed below. These are the main features of the product in more detail result is described as follows.

4.1 TOOL FOR DESIGNING HAND GESTURES This is the most important deliverable of this project. This platform provides a tool for developing hand gesture based animation and as mentioned earlier that it is targeting naïve user such as dance scholar and dancers so Interface is simple and understandable. Whole interface can be divided in to three parts one is the Animation Arena second is the user control for developing hand gestures and animation bar to handle the animation timestamp.

4.2 HAND GESTURES DESIGNED WITH THE HELP OF THE TOOL Some designed hand gestures are Original dance step

Synthesized hand gesture

SHIKHAR

Technical Report, Department of Computer Science, JIIT

•

9

SINHAMUKH

MAYUR Fig 3.4.2 Some designed hand gestures 4.3 ANIMATION The main feature of this tool is to develop Hand Gesture based animation. And this is the most exciting work and with the help of the tool it can be done very easily. For designing an animation user simply perform any gesture on either part of the hand and that gesture will be stored and distributed over animation, user can add another animation by selecting other part of the hand and it will keep on adding in the animation. Here is one example of numbering from one to five which is stored in the script file.

Fig 3.4.3.1 Initial Hand model 4.4 ANIMATION SCRIPT FILE Portability issue and size issues are handled by storing the script of the animation. XML based script file is used to store the information of the animation. Prototype of the XML file is given below.

10

•

S.GOEL, P. SINGH,

et al

Direction in integer ROTATE / TRANSLATE Continuous Occurrence

Fig 3.4.3.1 Script file prototype As mentioned in above prototype there are tags for each part of the hand and for each part of the hand there is DATA tag which stores the information of the transformation like type of the transformation, number of

Technical Report, Department of Computer Science, JIIT

•

11

time of the transformation and direction. There are multiple entries of DATA tags are possible in each portion of the tag except last tag. In the last tag GESTURETEXT information or subtitle are stored.

4.5 ANIMATION PLAYER This tool is a compact form of the whole software which can be used for only playing gesture animation based on the Script file. The purpose for deigning this tool is to provide compact player to those people whom there is no requirement of designing only they want to play and see the animation with script files.

4.6 COMPONENT FOR DESIGNING HAND GESTURE Another component is designed for those who want to develop application based on these gestures. They have to simply drag drop this component and merely by writing few lines hand gesture animation will be started and it is size independent and user can give script file to play the animation.

Fig 3.4.5 COMPONENT FOR DESIGNING HAND GESTURE 5. CONCLUSION The tool for developing Human Hand Gesture synthesizer has been successfully implemented. With the help of this tool we can develop hand gesture which can be used in Indian Classical Dance, Sign Language or in Pictographic Symbols as result has been shown in previous section. The main features of this tool as is to devise hand gesture based animation for Indian Classical Dance. Using this tool Dance Mudras, Sign language symbols, pictographic symbols any type of hand based modeling can be developed. For the portability and reducing size of the animation script based animation is used, user can store the animation in the form of XML file and XML file can be used to play the animation. Database of the sign language symbols, dance Mudras and pictographic symbols are given so that for teaching instead of designing direct

12

•

S.GOEL, P. SINGH,

et al

database can be used. Along with the animation designer can also add subtitle so that it become very easy to use this tool for teaching and demonstrating sign language or dance steps.

6. FUTURE SCOPE There are vast fields where we can use this as component for developing different hand gestures. Learning hand gestures will be more precise and easy if these are designed with the help of the gloves or vision based techniques so input method can be more sophisticated. Hand gesture movements which are used in dance have particular mapping of songs with particular Mudras, software can be designed with the help of this tool in which on the basis of category of Indian Classical Music, Mudras can be played. Apart from hand motion of all the body part can be designed for dancers and it will be more beneficial if by capturing dance in video form can be replicate in the animation. One extension of this project can be designing hand gestures with the help of text or sentence which will be input.

7. REFERENCES [1] H, Thomas and P, Vladimir. “Hand Gesture Modeling, Analysis and Synthesis”. University of Illinois [2] Horace H. S. Ip, Sam C. S. Chan, and Maria S. W. Lam, “Hand Gesture Animation from Static Postures Using an Anatomy-Based Model”, IEEE [3] George ElKouraand Karan Singh, “Handrix: Animating the Human Hand”, The Eurographics Association 2003. [4] CyberGlove - Virtual Technology. http://www.virtex.com/ [5] GloveGRASP(TM) - HITLab package for SGI computers. http://www.genreality.com/ [6] HumanGlove(TM) - A 20. http://www-hmw.caribel.pisa.it/ [7] Horace H S Ip, Maria S W Lam, Ken C K Law, Sam C S Chan, "Animation of Hand Motion from Target Posture Images Using an Anatomy-Based Hierarchical Model", Computers & Graphics, vol. 25, no. 1, pp. 121-133, 2001.

Technical Report, Department of Computer Science, JIIT

•

13

[8] Horace H S Ip, Sam C. S. Chan and Maria S. W. Lam, “Hand Gesture Animation from Static Postures Using an Anatomy-Based Model”, Proc. Computer Graphics International 2000, Geneva, Switzerland, pp. 29-36, June 19-24 2000. [9] Horace H S Ip, Sam C. S. Chan and Maria S. W. Lam, “Hand Action Coding System for Hand Gesture Simulation”, Proc. of International Conference on Image and Graphics 2000, Tianjin, China, pp. 626-629, August 16-18 2000. [10] Horace H S Ip, Sam C. S. Chan and Maria S. W. Lam, “HACS: Hand Action Coding System for Anatomy-Based Synthesis of Hand Gestures”, Proc. 1998 IEEE Int’l Conf. Systems, Man, and Cybernetics, San Diego, California, USA, pp. 1207-1212, Oct. 11-14 1998. [11] XML Family, http://www.xml.com/pub/a/2003/10/08/family.html [12] XAML in Windows Vistas, http://msdn.microsoft.com/windowsvista/experience/ [13] A. Torige and T.Kono. "Human-interfacace by recognition of human gestures with image processing recognition of gesture to specify moving directions", IEEE, 1992 [14] J. Kuch and T. Huang, "Vision based hand animation", Springer 1993 [15] R. M. Sansa and D. Thalman, "A hand control and automatic grasping system for synthetic actors", Eurographics Association, 1994 [16] T.E. Starner and A.Pentland. "Visual recognition of american sign language using hidden markov models", IWAFGR, June 1995 [17] VRML , http://en.wikipedia.org/wiki/VRML, http://www.w3.org/MarkUp/VRML/ [18] H. Nishino, K. Utsumiya and K. Korida ”3D Object Modeling usin spatial and pictrographic gesture” published in ACM VRST 1998. [19] H. Nishino, K. Utsumiya, D. Kuraoka and K. Yoshioka ”Interactive two-handed gesture interface in 3D virtual environment” published in ACM VRST 1997. [20] Andrea Corradini ”Collabroative Integration of Speech and 3D gesture for masp-based application” published in ICCS 2004.

14

•

S.GOEL, P. SINGH,

et al

[21] K. Hyun Ho, Y. Kuno and Y. Shirai ”Context based recognition of manipulating hand gesture for human computer interaction” published in 1998 in springerlink. [22] Mudras used in Indian Classical dance http://www.somadance.in/index.htm [23] Types of mudras which are used in Vedas http://health.indianetzone.com/yoga/1/types_mudras.htm [24] Kathak Mudras http://www.gaurijog.com/Mudra/Mudra.htm [25] Virtual human markup language http://www.vhml.org/ [26] Neumann Gustaf and Strembeck Mark ”Design and implementation of a flexible RBAC-Service in an Object-Oriented Scripting language” published in CCS’01 November 5-8 2001, Philadelphiya USA. [27] Reynolds W. Craig ”Computer Animation with scripts and Actor” published in ACM. [28] Body Animation Markup Language http://vrlab.epfl.ch/research [29] Octaga Player, http://www.octaga.com/ [30] Flux Player, http://www.mediamachines.com/ [31] BS Contact, http://www.bitmanagement.com/products/bs_contact_vrml.en.html [32] Cortona VRML Client 5.0, http://www.parallelgraphics.com/products/cortona/ [33] Xj3D, http://www.web3d.org/x3d/xj3d/ [35] H3D, http://www.h3d.org/ [36] CyberX3D, http://www.cybergarage.org/vrml/ [37] Web3D Consortuim, http://www.web3d.org/ [38] X3D Specifications, http://www.web3d.org/x3d/specifications/