> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <

1

A Methodology for Evaluating E-Learning Impact in Students of Distance Education Cardona R., D.M, Sanchez, J.M, Ph.D.

Abstract— This paper presents background information on the impact assessment which shows the influence of distance education and its relationship to e-learning, showing the construction of a methodology to assess the impact of the learning process in students of distance education. The proposal methodology in this paper is interesting since studies show not only a way used to address impact assessment, and most of them focused on assessing the technology platform and very few evaluate the impact on students by the use this new form of learning. Index Terms— Impact, e-learning, distance education, learning process, methodology.

I. INTRODUCTION

I

recently years Information Technology has an effectual roles in academic learning, this has allowed people from different economic, social, academic, access to training through virtual learning solutions known of e-learning, however, in many cases these deployments have led to positive and negative impacts for the various actors in the learning (students, tutors and institutions). While it is important institutions and their teachers, are more importantly their students and their learning processes carried out inside an institution, for instance is necessary to do the assessment, specially when they are immersed in an educational model that requires responsibility and independent study as distance education. N

Therefore, this paper proposes a design of methodology to assess the impact of the learning process in students using elearning as a training tool, validate whit a real case.

II. BACKGROUND A. Distance Education and E-Learning Distance Education is an ancient educative modality, which

D. M. Cardona is student master in system engineer and computer science of the National University of Colombia, work in the National University Open and Distance (e-mail: [email protected]). J. M. Sanchez is a PhD in information and communication technologies work in the systems engineers and industrial department, is investigation advisor the National University of Colombia (e-mail: [email protected]).

is an alternative model, show relationship and pedagogical and didactical rationality whit knowledge and learning, between medium and mediation, between time, space and context, between formation and work, and between local, regional and global. [23] Your fundaments is the discipline and responsibility of students, because send you by mail study documents and he return solver the exercises [20], now whit the Technology advance, the not send traditional mail, because the students Internet access, consult their e-mail where the tutor has been sending information or visit web pages participate in talks and question on study information, so in the education distance the participants stay in different locations. In distance education the figure of the teacher is not as marked as in education courses, and the teacher comes to be the guide and adviser in the process of student learning, the latter considered the heart of education because constructs knowledge from the reality presented, using the constructivist and collaborative approaches. [11] The constructivist approach encourages critical thinking, constructive and creative during the learning process, understanding that human beings actively construct their knowledge based on what we know and an active relationship with the knowledge of those with whom they interact. [24][25] For collaborative learning is any activity in which two or more people working together to define a meaning, explore a topic or improve skills in which each group member is responsible for their learning and that of the rest of the group. [24] Defines e-learning as a learning system through the use of electronic media and is presented as a system as flexible, adaptable to different domains and modalities. [10] [27] In this sense, e-learning has allowed a new model of education where the physical space is replaced by the virtual space can have an individualized education system, flexible, resourcebased, accessible, regardless of distance, interactive, costefficient, updated with sophisticated educational tools, with massive access to information, with new roles for both teachers and students, and especially for the development of new skills. [10] [28] 1) Principles of e-Learning These are the principles, which institutions should apply to enrich the educational experience of e-Learners and to ensure that courses supported by appropriate information resources

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < [32]: • • • • • • • •

Responsibility Quality Equity Access to information and help Information literacy Integration Communication Whole-process costing

2) Elements of e-Learning There are several elements of e-learning: • Self-paced training delivered over the Web -- or via book, CD, or video. • "Virtual events," which could take place in a virtual classroom, virtual lecture hall or expert-led online discussion. Mentoring -- which might entail coaching, a help desk, office hours, and periodic check-in or email exchanges? • ”Simulation”, because we learn by doing • Collaboration -- either joint problem solving or discussion among study groups, real-time or in a discussion forum in which postings can come at any time • Live workshops, some topics still best taught in person by an instructor or expert B. Evaluate Process The concern for evaluating e-learning is leading to important initiatives and experiences to establish global standards for certifying its quality [12] The objective focuses on specific search criteria and indicators to respond to questions which is assessing the quality of training in specific environments, with specific means. [12] Because it is necessary to evaluate the assessment can provide a framework for thinking about what are the weaknesses and help to any change in practice has benefits, like the results lead to action to improve teaching and learning the institution. [31] Several assessment models such as Kirkpatrick model proposed in 1994 and proposed by Steven J Mc Griff in 2000, measures used to evaluate traditional formats, Steven suggested that evaluation should be performed: before starting the implementation of the technological innovation, called needs assessment during the development and implementation (pilot) called formative assessment, and after a course that used the technology, called summative evaluation, which allows monitoring and there are changes in teaching and learning the behaviors and achievements. [12] C. Some studies Impact of e-learning In the study by Cuevas et al. [22] showed the results obtained by evaluating the impact of a technology platform used in the Mexican Institute of Technology University of Sonora (ITSON), which assessed the academic performance of students and the perceptions of various stakeholders regarding the utility e-learning. Where the results showed that the use of

2

the platform does not positively affect the academic performance of students, however, the activities offered through SAETI (System to Support Education with Internet Technologies), to complement activities traditional not affect performance in a negative way. At the University of Malaysia [1], the directives of the university decided to implement e-learning because of the low cash cost, high reliability and efficiency, cooperative and easy to use, where they found that coordination through the cooperative work provides a greater contribution to the members of the university, however, after having implemented e-learning members of the university had to face problems such as inadequate infrastructure, the network drops, maintenance costs, among others, which caused great impact in the academic community. In the study of Ga-Jin made in 2007 [4] is a methodology for evaluating e-learning tools, where the first thing that does is get the key activities for the e-learning such as: operations manager, content manager, manager, IT (Information Technology), manager of training and administration of culture, then identifies factors of e-learning for each activity and each factor determining the indicators to be measured, the study concluded that the quantification of the measurement was difficult because each subject had different units of measurement, so to use a tool monitoring of learning is necessary a standardized set of indicators. In the case of the University of Croatia [2], a study was conducted to assess the impact of using the platform at Angel and graduate students which identified 17 indicators to measure them are: (1) educational background of the participant, (2) level of computer training (3) types of participants, (4) their learning style and intelligence, (5) obstacles visible in the e-learning (e-learning barriers), (6) attention, (7) content (suitability, format preferences), (8) design education (9) specific organization (10) Preferences the logistics of e-learning; (11) preferences for the design of elearning (12) technical capabilities available to the trainers (13) collaboration (14) accessibility available to trainers (15) motivation (16 ) attitudes and interest, and (17) return self (the sense of participating in the effectiveness of e-learning environment). Which concluded that these indicators should be the starting point for the development of any e-learning initiative, and that each e-learning initiative is unique, so these indicators should be modified or adapted. In addition to this study in the School of Business Administration from the University of Northeast China in research conducted by Liu [18], shows that it is important to know the satisfaction level of students taking a course via the internet, as satisfaction is a measure attitudes that can affect intentions to adopt this method of learning or not. Mandinach in 2005 [13] proposed a method for effective elearning, due to the impact on teaching and learning activities in their paper indicates the nature of e-learning instructional and assessment methodologies discusses how to analyze begins the e-learning affect teaching and learning process.

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < These studies and others [14] [15] [16] [22] show that assessing the impact of e-learning is important for educational institutions, enabling them to make significant decisions, however, the various indicators used in the studies are not standardized [12], i.e. each assessment initiative considers indicators according to their context and in some cases in the same study units of measurement are different, hinders their

Fig. 1. Sub-characteristics of each characteristic.

analysis [4], similarly, the indicators should not be general and as each initiative is unique and has e-learning particulars evaluation indicators should be adjusted to the particular case study. [2] [3]

III. METHODOLOGY APPROXIMATION In this section we show a first approximation of methodology for assessment the impact on student learning process in a distance education. Step 1: ´First we determinate the characteristic to assessment, we divide in characteristic of solution e-learning functionality, reliability, usability, efficiency, maintainability and portability, where is divide by sub-characteristics how see in figure 1., characteristic e-learning course (A1-A3, and B1B10), aspect of learning (daily activity, errors that appear in the construction of concepts, Development of attitudes, Time it is use, use of different materials), and competences (interpretative, propositive and argumentative). The detailed description of characteristics e-learning course is following:

A: Contents Evaluator Following three characteristics are checked by independent Content Evaluator. A1: Course content meets the requirements for textbook The lessons include clear objectives, study texts are enhanced with assignments and/or case studies, at the end of lessons summaries, check answers, literature, webliography and other sources are available. A2: Course content on satisfactory professional level

3

The content is evaluated by expert in particular field of study. A3: Course content for full semester study The lessons in online course typically correspond to lectures in full-time study, on the other hand, assignments, case studies and tests usually match exercises or practices. B: e-learning Evaluator Following ten characteristics are checked by independent elearning Evaluator. B1: Course general information The course information should include course title, contact addresses and profiles of course instructors, course code, field of study, number of credits, course syllabus including an clear statement of the roles, responsibilities and expectations of the instructor and students, technical requirements including links to tutorial modules for any additional skills that maybe required, and a statement on relevant institutional policies as late assignment, plagiarism policies, etc. B2: Course calendar with event and activities The calendar represents a time schedule of lesson’s topics readings, assignments, tests and exams including all deadlines. It can inform about course events and activities, e.g., date of virtual classrooms, personnel meetings, consultations, etc. B3: Course lessons with clear structure and motivational components The comprehensible lessons with well-defined structure including objectives, keywords and lesson summary, links to self-evaluated tests and assignments, list of literature and resources. Students should be engaged through humor, novelty, game or surprise elements, testing, adventure. etc. B4: Adequate level of design and aesthetics. Simplicity is the key. B5: Appropriate rnultimedia support of lessons The quality pictures, animations, audio and video files are needed. Note that use of multimedia depends on course nature. i.e., for some courses multimedia files are not required. On the other hand, audio/video streaming of lectures is usually useful. B6: Keyword and glossary of term At least one-two keywords :for every slide of lesson ate recommended. B7: Assignments and discussion forum The assignments usually correspond to exercises in full-time study. Discussion forums snppon communication within elearning course. Information about assignments and discussions should be defined in course calendar as well. B8: Evaluation systems including self-evaluated, current and final tests Every lesson should be linked with at least one selfevaluated test to ensure a study feedback. B9: Additional resources Additional resources include a list of literature and reading materials, webliography, FAQ, case studies, references and other links to learning resources. B10: Course text ready to download and print

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < As many students are still grounded in traditional “paper and pen” methods, a good idea is to have pages available in standard print formats for quick downloading and offline use. Step 2: Design a questionary for collecting data from students, who are using e-learning

[9]

[10]

Step 3: Assign points, which are obtain from questionnaires in step 2. [11]

Step 4: Assign the value for each sub-features.

[12]

Step 5: Determine the population to evaluate and apply questionnaires [13]

Step 4: Compute the value of characteristic. [14]

IV. FUTURE WORK This is a first approximation of the construction of the methodology, there is theoretical perform validations. To validate the model of an objective is intended to apply to a real case study, so as to allow feedback on the results of the proposed methodology

[15]

[16]

[17]

V. CONCLUSION E-learning evaluation needs to be planned meticulously and designed at an early stage of the training or teaching and learning programme. Moreover, e-learning evaluation will enable the researcher to answer the research questions and meet the objectives of the study. Perform a methodology for assessing the impact of the elearning in distance education is necessary, since in various impact assessment studies, using different ways carrying out evaluations.

[18]

[19] [20]

[21]

REFERENCES [1]

[2]

[3]

[4]

[5] [6]

[7]

[8]

Divjak, N.B. Imaginative acquisition of knowledge – strategic planning of E-learning Information Technology Interfaces. 28th International Conference on, Information Technology Interfaces, 2006, 47-52. Fetaji, B., Fetaji, M. E-Learning Indicators Methodology Approach in Designing Successful e-learning. 29th International Conference on, Information Technology Interfaces, Cavtat Croatia, June 25-28, 2007. pp. 307-312. Fetaji, B. E-Learning Indicators Approach to Developing e-learning Software Solution. The International Conference on, “Computer as a Tool”, EUROCON, 2007. 2687-2694 Ga-jin, I. Developing Evaluation Tool for e-Learning Management of Engineering an Technology, Portland International center for, Management of Engineering and Technology, 2007, 1526-1537. Colace, F., A. P. Evaluation Models for E-Learning Platform: an AHP Approach. Frontiers in Education Conference, 36th Annual, 2006,1-6. Fang, C. H., T. M. C. Study on Effects, Limits and Current Situation of E-learning System an Example on Small-median Enterprises Industrial. IEEE International. Contreras, W. E. Evolución de las aulas virtuales en las universidades tradicionales chilenas: el caso de la universidad del bío-bío. Revista Horizontes Educacionales, Vol. 12, Nº 1: 49-58, 2007. Peña, M., Avendaño, B. L. Evaluación de la implementación del aula virtual en una institución de educación superior (Fundación

[22]

[23]

[24]

[25]

[26]

4

Universitaria Konrad Lorenz, Colombia). Revista Suma psicológica, Vol 13 No 2. Septiembre 2006, 173-192, ISSN 0121-4381. Colombia, Departamento Nacional de Planeación, Ministerio de Comunicaciones, DNP: UINFE-DITEL (2000). Documento Conpes 3072, Agenda de Conectividad. Bogotá, D.C. citado en [10] Malaver R, F.; Observatorio Colombiano de Ciencia y Tecnología. Libro Vigilancia Tecnológica y competitividad sectorial: lecciones y resultados de cinco estudios, ISBN: 978-958-44-1156-3 Primera edición: Bogotá - Colombia, 2007, recuperado el 20 de febrero de 2009, disponible: http://www.sena.edu.co/downloads/2007Portal/Innovaci%C3%B3n%20 y%20 competitividad/Vigilancia_tecnologica_CCB.pdf Cardona, G. Artículo tendencias educativas para el siglo XXI, educación virtual, online y learning elementos para la discusión, 2002 Rubio, M. J. (2003). Focus and models of evaluation of the e-learning. RELIEVE, Revista Electrónica de Investigación y Evaluación Educativa. v. 9, n. 2, p. 101-120. http://www.uv.es/RELIEVE/v9n2/RELIEVEv9n2_1eng.htm Mandinach, E. B., Teachers College Record Volume 107, Number 8, August 2005, pp. 1814–1835 Copyright © y Teachers College, Columbia University 0161-4681 Olds, B. M. Effective Strategies to Assess the Impact of e-Learning, procceding e-Technologies in Engineering Education: Learning Outcomes Providing Future Possibilities 2002, Vol. P1, Article 24, Davos, Switzerland Malataras, P.; Pallikarakis N. Evaluation of an E-learning course in Biomedical Technology Management. Twentieth IEEE International Symposium on Computer-Based Medical Systems (CBMS'07) 0-76952905-4/07 © 2007 Yusminar Yunus, Juhana Salim Framework for the Evaluation of Elearning in Malaysian Public Sector from the pedagogical perspective Universiti Kebangsaan Malaysia. Rainer, K., Klaus, J. Towards an Evaluation of (e-) Learning Systems. American Association for Artificial Intelligence (www.aaai.org). Germany 2005 Liu, Z. Learning from E-commerce from E-learning Information Technologies and Applications in education, ISITAE ’07. First IEEE International Symposium on, 2007. 193-197. Louiza, A., Restrepo, L. G. An Experience in the Evaluation of elearning for it training and certification Lobato, J. R. Aplicaciones multimedia para la formación del siglo XXI: nuevos medios, motivación y evaluación, Universidad de Málaga, congreso CEIIE, Congreso la Educación en Internet e Internet en la Educación, 2001, disponible en: http://congresos.cnice.mec.es/ceiie/area2/documentacion/comunicacion es/html/2comunicacion14.html, recuperado el 20 de febrero de 2009 Hellers, N. Impacto del e-learning en las gerencias de ventas. 2005. Recuperado el 19 de agosto de 2005, de http://www.elearningamericalatina.com citado en [10] Cuevas S., Omar; García L, Ramona; Cruz M, Isidro. Evaluación del Impacto de una Plataforma Tecnológica utilizada en una Universidad Mexicana, Encuentro de Educación Virtual, VirtualEduca. Disponible en línea: http://www.virtualeduca.info/ponencias/24/Ponencia%20Cuevas_Garcia _Cruz.doc. Recuperado el 20 de febrero de 2009 Comité Institucional de Autoevaluación de la UNAD, Informe para Verificar condiciones iniciales a fin de ingresar al sistema nacional de acreditación, Ediciones Hispanoamericanas, Bogotá, D.c., junio de 2008. Chinchilla, A. Material de estudio del Módulo Pedagogía del Trabajo Colaborativo en Ambientes Virtuales de Aprendizaje del diplomado tutor de curso virtual de la UNAD, junio 2008. Sánchez-Cortés R. Sosa, García Manso A., Sánchez Allende J., Moreno Díaz P., Reinoso Peinado, A. J. B-Learning y Teoría del Aprendizaje Constructivista en las Disciplinas Informáticas: Un esquema de ejemplo a aplicar. Recent Research Developments in Learning Technologies. Third International Conference on Multimedia an Information & Communication Technologies in Education. 2005. España. Disponible en: http://www.formatex.org/micte2005/AprendizajeConstructivista.pdf Fandos, M. y González, A. P., Estrategias de Aprendizaje Ante las Nuevas Posibilidades Educativas de las TIC. Third International Conference on Multimedia an Information & Communication

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <

[27] [28] [29]

[30]

[31]

[32]

Technologies in Education. 2005. España, disponible en: http://www.formatex.org/micte2005/227.pdf Bates, T. Technology, e-learning and distance education. Second Edition. Ed. Routledge Taylor & Francis Group. 2005. Tudesco, A. B. Notas Educación a distancia y E-learning: Ideas previas para el debate, UNED. Argentina 2005. Comezaña, O. & García, F. (2005) “Plataformas para educación basada en web: Herramientas, procesos de evaluación y seguridad”. Informe Técnico, Universidad del Salamanca p. 66. Disponible en: http://tejo.usal.es/inftec/2005/DPTOIA-IT-2005-001.pdf citada por William Contreras en [7] Chiarani, M.; Pianucci, I.; Lucero, M. Criterios de Evaluación de Plataformas Virtuales de Código Abierto para Ambientes de Aprendizajes Colaborativos. Anales del VI Workshop de Investigadores en Ciencias de la Computación (WICC 2004) ISBN: 950-665-337-2. Neuquén, Argentina mayo 2004. Disponible en: http://www.dirinfo.unsl.edu.ar/~profeso/PagProy/articulos/wicc_2004_1 .pdf. Consultado: Junio 2008. Shaw, R., Jen, H. Evaluation Cookbook. TLTSNConsultor. Glasgow University – Learning Technology Dissemination Initiative. Marzo 26 de 1999. ISBN 0 9528731 6 8. Disponible en: http://www.icbl.hw.ac.uk/ltdi/cookbook/ Elnaz, K.; Mohamadreza, M.; Ali, M. Evaluation of user satisfaction in center of e-learning-university of Tehran. In proceeding IEEE Computer Society. 2008

5