SI622 WINTER 2013 EVALUATION OF SYSTEMS & SERVICES: ASSIGNMENT 8
USABILITY TESTING Wr i t t enf orcl i ent ,Aci asLLCandsyst em Apr18,2013
GROUP0 5T EAMQI VE CHI NJUICHEN MOHAMEDGULAI D CHADKI JEWSKI HSI AOCHI HLI N SUNGJI NNAM
SI622 WINTER 2013 EVALUATION OF SYSTEMS & SERVICES: ASSIGNMENT 8
USABILITY TESTING Wr i t t enf orcl i ent ,Aci asLLCandsyst em Apr18,2013
GROUP0 5T EAMQI VE CHI NJUICHEN MOHAMEDGULAI D CHADKI JEWSKI HSI AOCHI HLI N SUNGJI NNAM
Table Content: Executive Summary
2
Introduction
3
Methodology and Scope
4
Findings & Recommendations
5
Discussion
10
Conclusion
11
Appendixes
12
Appendix A. Introduction Script
13
Appendix B. Scenario & Tasks
14
Appendix C. Pre Questionnaires
16
Appendix D: Post Test Questionnaires
17
Appendix E: Pre Post Questionnaires Results
18
Appendix E. Blank Logging/Transcripts Form
24
Appendix F. Notes Card for Testers
25
Appendix G. Informed Consent Form
26
Appendix H. Environment Preparation
27
Executive Summary This report documents the findings and recommendations that our team identified from conducting of usability test on Qlovi. Qlovi is a web-‐based application designed to encourage and enhance learning through reading in a primary schools. This study was conducted in order to develop a better understanding on how novice users would use Qlovi’s main functions. These tasks impart the core features of the application. We followed the standard UX procedures of testing usability by conducting usability in a lab like setting, carrying out pre and post test questionnaires and conducting short qualitative interview with our testers after performing tasks. Qlovi has two different interface designed for different users group. We evaluated the interface designed for teachers, which has more functionality than the students’ interface. Relatedly, recruiting k-‐ 12 student proved to be challenging and time consuming. To satisfy the scope and the breadth of our study, our team recruited users who have teaching experiences and within diversified demographics and background. We reached out via our personal and professional connections, and selected a total of five viable participants. We conducted five in-‐depth usability tests on these users; the test includes five main tasks. These tasks reflect the typical workflow of English teachers in primary schools: assign a reading and conduct assessment related to the readings. Our team members conducted the tests in pairs, and subsequently gathered to analyze data of each test. We consolidated our major findings and produced findings and recommendations. Here is a list of our major findings. The presence of certain navigational and interactive features is not clear to users. Users ignore 'Help' functionality, even when lost or confused about the system. System feedback is missing, misleading, or confusing to users. Navigation architecture is not optimally designed to allow fluid experience moving between pages and tasks. Indications of important information to help users make selection are missing. For example, information about books like date of publication, reading levels, and compliance with core requirement are missing.
Introduction Evaluated System This report conducts an analysis of Qlovi, a web-‐based application that is currently in development by Acias LLC. The overall purpose of Qlovi is to encourage, enable, and enhance learning through reading in a primary school environment. Qlovi aims to achieve its purpose by serving as an electronic book (eBook) repository for its target populations. It has arrayed itself to cater to two user groups: teachers and students. In regards to the former, teachers will use the system to identify appropriate literature available within Qlovi’s library for assignment to students. Students will then read the specified material to answer questions or provide feedback, which will demonstrate their comprehension. In regards to the latter, students use the system to identify literature of personal interest within Qlovi’s selection to consume on their own. To enable these uses, Qlovi aims to incorporate an intuitive user-‐interface that emphasizes choice and accessibility for its student users while simultaneously providing teachers the ability to obtain feedback and measure student comprehension of the assigned material. The system incorporates a number of functions and features that enhances its performance in these roles. This includes the integration of a recommendation service that allows individual students to find books of interest as well as tracking analytical information regarding student performance and activities.
Study Purpose & Motivation The culmination and most practical segment of a usability examination is the usability test—that is, observing users as they navigate the system in an attempt to complete a number of pre-‐defined tasks. The ability to view and review the performance of users as they interact with the system is valuable; it provides a framework of what works and what doesn’t; where and when there are breakdowns between the user and the system; and insight into how users accomplish tasks, whether it via the designer’s intended way or an alternative path through the system. With this motivation in mind, our team decided on the following five primary questions that we would like to answer through this usability test: How well would Qlovi fit into a realistic teaching environment? How do users from different age groups interact with Qlovi’s interface? How easy is it for users to perform Qlovi’s core functions? Is the flow of activity while completing tasks smooth or disconnected? Do users effectively, or actively engage, Qlovi’s ‘help’ functionality?
Study Methodology
For this phase of the usability testing process, our team evaluated five users as they navigated through Qlovi in an attempt to accomplish five pre-‐defined tasks. These tasks were crafted in such a way as to focus on what we believe to be the basic and core functions of the system. After all users had completed the usability test, our team gathered to discuss the results. From these, we identified five key findings and developed associated recommendations for Qlovi’s development team to consider, which is available in the following section of this report.
Scope of Study Qlovi has two primary user types—teachers and students—and as such has created two different perspectives to cater to each group. To better focus our analysis, our team decided to continue to evaluate Qlovi through the teacher’s perspective; this interface is more complex than the student version due to the additional functionalities available to the user, such as module formation, personnel administration, and assessment creation. Additionally, Qlovi is riddled heuristic problem in some functions like grading assessment. We decided not evaluate advance functions with obvious usability problems.
Target Population
In order to satisfy our scope of study, our team focused on recruiting individuals that were either current teachers or those that had past teaching experience. We also focused on diversifying the backgrounds and demographics of our participants—whether it be age, gender, or level of teaching experience—in order to ensure that a wide spectrum of opinions were represented in our feedback.
Subject Recruitment & Characteristics Individual team members reached out via personal and professional networks to identify viable participants. Due to the time requirements of the project and limited resources available to the team, we focused on recruiting a total of five potential users. Of the five individuals that were recruited, four had prior teaching experience. Multiple generations of teachers were represented, as subject ages ranged from 20 to 60. Here is an overview data of our users. Demographics Teaching Experience Computer Expertise 28, Male Yes 4 60, Male Yes 5 55, Male Yes 4 33, Male Yes 4 20, Female No 4
Test Procedures
Prior to initiating testing with our five recruits, the team successfully completed a pilot test with a willing participant. The feedback allowed us to identify a number of confusing issues with our tasks. To counter this, we refined our tasks and developed a ‘smart card’, which is a document we distributed to participants with the task written down for reference purposes. Two team members administered the test at a time and location convenient to the participants; one team member fulfilled the role of moderator while the second concentrated on taking notes. The moderator provided an introduction and orientation to the test, allowed the subject to interact with and orient to the system for no more than three minutes, and presented a consent form for the subject to sign. All subjects filled out a pre-‐test questionnaire and then were guided through a pre-‐generated scenario and array of five tasks. Subjects ‘thought aloud’ while completing tasks to allow team members to identify their thought process while interacting with the system. This audio as well as their on-‐screen actions were recorded using capture software that was pre-‐loaded on available testing computers. Upon the completion of the test, the subject filled out a post-‐test questionnaire and then was debriefed about their experience performing the tasks by both team members before being released.
Instruments Utilized and Physical setting We conducted the tests in conference rooms, where we utilized “minimal lab setting.” Subjects utilized a team member’s laptop to interact with Qlovi throughout the usability test. We projected Participants utilized Google Forms to fill out pre and post-‐test questionnaires. Based on the operating system of the device, we utilized Apple QuickTime (For Mac) or Adobe Flash Live Media Encoder (For PC) to record video and audio of the subject’s actions and voice throughout the test. These videos were then encoded and consolidated on YouTube for review by team members.
Tasks
As stated before, we decide to focus on main features of Qlovi and simulate workflow of a typical English teacher. Our comparison analysis and heuristic evaluation have helped us produce these tasks. We decided to focus on overall experience of using Qlovi and if the system can add value to teaching and enhance class management. Here is short description of tasks performed by testers: Level of Difficulty Tasks (Easy, Moderate, Difficult) Easy Locate and add an appropriate book for tenth grade students to the library Moderate Create a module for a class Difficult Modify and assign a module Easy Separate students within a class into two groups Easy Find a specific book in the Store and add it to an existing module
Findings & Recommendations
Finding #1: The presence of certain navigational and interactive features is not clear to users.
Evidence: A number of our test participants consistently missed navigational features within Qlovi’s interface. For example, the first task required the user to find a book that they determined to be suitable for their tenth grade English class. Users would consistently log in and encounter the ‘class’ screen; at this point they would normally navigate through the class interface, frustrated as they were unable to find where books were located within the system. It routinely took users two to four minutes to identify that the primary navigational bar was an interactive feature (Figure 1).
Figure 1: Subjects would routinely focus on the class navigation bar (orange), unaware that the primary navigation bar (red).
Figure 2: The book facet filtering bar that our users ‘did not notice’.
Additionally, once users had successfully navigated to the ‘Store’—the location to preview and select books—most began clicking through the book covers to find more information about their content. The task required users to find books appropriate for their tenth grade class. There is a navigational feature on the left side of this screen that allows them to sort books by age and difficulty that users did not engage (Figure 2); during debriefing, most users replied that they did not notice this bar.. Recommendation: Qlovi’s design team should take action to ensure that the interface’s navigation features and bars are more prominently displayed. In the case of the primary navigation bar, users may have routinely missed it due to the color of its inactive links, which neatly blend in with the page’s background color. In the case of the book filtering bar, its placement to the left of a large banner is the likely reason it was ignored and largely underutilized throughout our usability test iterations. Regardless if it is ignored be due to lack of color, emphasis, or placement, a navigational feature loses its ability to be an effective tool within the interface if it is largely invisible or unknown to the user.
Finding #2: Users ignore 'Help' functionality, even when lost or confused about system functionality. Evidence: Sometimes, participants were puzzled while conducting some tasks. But none of them accessed Qlovi’s help page to find the answer to solve their wandering. Users can access help page by clicking a menu that located in top navigation, along with other important menus like classes, modules, library, and store. However, when participants could not perform task in timely manner, they chose to discover the solution by their own or ask moderator, rather than accessing the help page. Also, participants who have accessed the help page during debrief phase said its text only form does not intrigue users to access.
Recommendation: Since Qlovi has many features that allow users’ interaction, it can be infer that help function also need to be attached closely with these functions. Accessing separate help page to find out more about certain function takes longer way to travel and time. We recommend Qlovi to make help features accessible around the actual interactive functions. Question mark icon near the function or mouse over popup for each menu can be a good way to locate user support content. Also, we recommend implementing more graphical or video for contents in help page, which is currently heavily relying on text, in order to provide users more intuitive support.
Finding #3: System feedback is missing misleading, or confusing to users. Evidence: Visual feedback within Qlovi’s system lack of visibility or missing. This leaves the users uncertain about the system status, or causing them to double check what actions they have taken. One of the testers kept double checking what he did after the performing the tasks. For example, after adding a book to a module from Bookstore, the dialog of the popup box showed “This book was added to your module and personal library” (Figure3). However, The user only added the book to module and not the library. It is not clear to the user the relationship between library and module function. Relatedly, users may have not want to have to book in library but in a specific module.
Figure 3: The notification showed [book] is part of your library, instead of [the book ] is added to your module and personal library, could be confusing to users.
Another confusion occurs frequently is that users are unclear about whether a module has been assigned or not. They depend primarily on the briefly appeared yellow notification more than the green “assigned” label, which is considered fainted and not enough contrasted (Figure 4,5). Furthermore, one of the users expressed that he did not have the feedback to know whether the book he just added to an assigned module would be unassigned or assigned. Users have to go back module to verify if selected book has been added to their library.
Figure 4: Low contrast “Assigned” label, an essential system status indicator, is not easy to be noticed by users.
Figure 5: ”Assign” button is high contrast and dominates the screen, causing the “assigned” label to be easily ignored by users.
Thirdly, a visual feedback which indicating the system is processing queries is missing when clicking on “Start Reading”. There was delay when one of the users clicked on ”Start Reading.” It took Qlovi about two second to preform the tasks, but there was no visual feedback on the interface that the system was processing the query. Thus, user proceeded to click on another button. Recommendation: First, the design team of Qlovi should make sure the notifications and the term used are in consistence with the system status so that users would not be confused by the notifications. Second, the design team can consider incorporate a processing gif whenever the system processes a query; In addition, the processing gif or animation should be place within the Qlovi’s interface to make sure that it is close to users’ visual focus. Thirdly, the labels indicating significant status and feedback should be clear and with high contrast. The design team can balance the color and size of labels and buttons. As shown in the Figure 5, currently the “Assign” button dominates the screen that leads users easily to ignore the “Assigned” label; also, if the module is already assigned, the “assign” button should be disabled, thus the system status would be more clearly conveyed. Finding #4: Navigation architecture is not optimally designed to allow fluid experience moving between pages and tasks. Evidence: The top navigation bar include following options: Classes, Modules, Library, Store and Help There is also sub navigation options in Qlovi that include Overview, Students, Groups, Grades, and Calendar. The relationships between top and sub navigation bar is not clear. Teachers cannot create or access saved modules from the top navigation, although there is module option in top navigation bar. Teachers have to choose class from the top navigation and choose module from sub navigation bar. This was confusing to our testers. Additionally, both navigations bars include the option “Module”, and it is clear what is the difference between two options for new users (Figure 6). Two testers faced major difficulties creating module because they access module from the top navigation bar and from bottom bar. Relatedly, the main home page as users login is the module, however, Qlovi is a reading platform more than anything. Books are not emphasized in the interface; users have to make few clicks in orders to
access books. One tester complained, “Books were not prominently displayed.” Qlovi interface starts with emphasizes module instead of book as the first task. However, typical books are assigned and quizzes come next.
Figure 6: Information Architecture is messed up…
Recommendation: Qlovi needs to establish effective and straightforward navigation architecture that reflect or correspond with workflow of a typical teacher in primary education. Qlovi markets itself as alternative to reading platform, however, the interface emphasis on modules. Additionally Information architecture should correspond with the need of typical English teacher. Tasks flow should start with books instead of a module. Reading come first and modules. Also, Best practices of of IA suggest having only on navigational bar; either vertical or horizontal1. Also a dropdown menu could help users figure flow of task better than having compartmentalized sub navigation bar. Additionally, the “Module” option should appear only once. There are two Module options and it is not clear what is the difference between them. We recommendation more standard navigation tabs like in Figure 7.
Figure 7: More straightforward navigation structure (Cameron Chapman, sixrevisions.com) 1
Cameron Chapman, http://sixrevisions.com/user-interface/navigation-design-patterns.
5: Indications of important information to help users make selection are missing.
Evidence: One tester commented when asked to locate a book for his tenth grade students, “The difficult is not finding the books but decide which book to choose.” The information provided for books don’t show important information to help teachers choose appropriate books. These following information are missing: target age, reading level, compliance with core requirement, and length of the book. Most of our testers failed to choose suitable book for requested class and ended up choosing a random one.
Figure 8: detailed information (reading level, core standard requirement) about books
Recommendations: There are many other ways Qlov can provide to help teachers choose books for their students. Qlovi can simply show more information about books when users click on book. Also Qlovi can allow teachers to review and rate books. Showing related topics or tagging is another way to provide assistance. In terms of incorporating with the Qlovi system, showing and sharing some module examples and the readings inside can help teachers to judge and facilitate communication among teachers.
Discussion This usability test gave us opportunity to evaluate the system based on outside viewpoint. It was a good chance to gather practical opinions about Qlovi from target users. However, this usability test still had following limitations. First, tasks were not able to cover every feature that Qlovi is trying to provide. Qlovi is still in developing phase. Like we found in heuristic evaluation, some of features in Qlovi lacked consistency, aesthetic design, and functionality. Although this usability test tried to simulate the realistic usage of Qlovi, in teacher’s perspective, Qlovi’s beta status restricted to test out every features. For example, grading feature, for quizzes taken by students, was not reliable. We found that sometimes, answers from students were sent to different class or some of them were not able to grade at all. Nevertheless managing students’ performance is one important feature of Qlovi, we could not include that in this usability test. Task design could be more sophisticated if the test was conducted after the system’s development.
Second, due to the limitation on resources, this usability test was conducted among only five participants. Although usability test relatively does not require many participants, compare to survey or other quantitative methods, the results from five participants may unable to generalize the usage of every prospective Qlovi users. Participant’s demographic was also limited within Ann Arbor area and University of Michigan. Third, this usability test was conducted based on teacher’s perspective. Qlovi has two distinctive targets: teachers and K-‐12 students. We chose to focus on teacher’s perspective since Qlovi offers more features to teacher group. As they are likely to face more technical difficulties with interface, it was easier to try various tasks for the usability task. Also, finding participants with teaching experience was easier than recruiting K-‐12 students. Along with the issue of small sample size, this is closely related to limited accessible resources, which may considered in future study. So far, across the semester, we have gathered findings and recommendations from using various methods. Interaction map helped us to understand the structure of Qlovi. We could deepening our understanding of prospective users by building use case scenarios based on interviews and personas. Survey provided us quantitative information on users. Comparative and heuristic evaluation helped to evaluate Qlovi’s weaknesses. Lastly, usability testing gave us outside opinions in practical usage. Our next step will be wrapping up findings from these methods and suggest overall recommendations for Qlovi.
Conclusion We conducted usability test of Qlovi with five participants who have teaching experience and vary in age and IT skills. We generated five tasks based on core functionality of Qlovi. Test subjects were introduced to the process, completed pre and post-‐test questionnaires, their on-‐screen performances were recorded and noted while conducting the usability testing. We shared and consolidated findings from each test, and we identified the key issues as follows: Finding #1: Navigational and interactional features are not obvious to users. Finding #2: Users ignore 'Help' functionality, even when lost or confused about the system. Finding #3: System feedback is missing, misleading, or confusing to users. Finding #4: Navigation architecture is not optimally designed to allow fluid experience moving between pages and tasks. Finding #5: The descriptions of books were not enough for users to judge whether they would choose to use or not. Qlovi is in the development phase. Consequently, a number of core functions are not fully available for usability testing. Our tasks assessed the usability of functions that varied from easy to moderate. Because of this limitation, this report does not offer insightful findings about advanced functions of Qlovi like grading and using analytics. Secondly, because of the limited time and resources, we recruited only five participants for the usability tests. Because of this small sample size, our findings result may not reflect on Qlovi’s broader target users. Finally, our evaluation has limited scope because of resources constrains. Our usability test only evaluated teachers’ interface. Thus, we cannot say that we have identified all the potential usability issues for the whole target population.
Findings from usability test support our findings in heuristic and comparison evaluation. Designers at Qlovi are aware of some functions limitation. For future evaluation we suggest recruiting current teachers and student to do usability testing on both interfaces. Additionally, we suggest fixing obvious bugs in the system before conducting major usability evaluation to better examine the experience of users and not function limitations.
Appendixes
Introduction Script: Qlovi Introduction Script: Good [Morning/Afternoon/Evening] and thank you for agreeing to participate in this study. We’ve enlisted your help to conduct an evaluation of Qlovi, which is an electronic reading system aimed to cater to the needs of both students and teachers in primary schools. In order to accomplish this I will be guiding you through a scenario and asking you to accomplish five tasks within the system. Your performance will help us identify where Qlovi excels or needs improvement. Please work through the tasks to the best of your ability. Also, indicate to me when you feel that you have accomplished the task or do not want to continue so we can move on to the next. If you’d like, I can give you a few minutes to orient yourself to the system at this time. Also, I have a questionnaire that I would like you to fill out as well.
[If desired, the user orients him or herself to Qlovi for about two minutes] [User fills out the pre-‐test questionnaire]
I want to stress that this is an evaluation of the system and not your performance. There is no ‘right’ or ‘wrong’; observing any difficulties you experience while carrying out these tasks are just as valuable as your ability to accomplish them, if not more. Our team and school are not affiliated with Qlovi or its development team in any way, so please feel free to share your feelings about the system throughout the study. Your candid opinions are valuable and will not hurt anyone’s feelings. Please ‘think aloud’ while you are working on these tasks. By ‘think aloud’, I mean audibly communicate what you are thinking; what you are doing, why you are doing it, and what you plan on doing next. This will allow us to better understand where there may be disconnects between the system and the user. While you are completing these tasks, please pretend that I am not here. You may ask clarifying questions as I assign tasks throughout the study if you don’t understand something, but I will not be able to answer any questions while you are working on a specific task. If you don’t know how to do something, you don’t know how to do something. That’s valuable information we’re after. Also, I would like to make you aware that your performance will be recorded, both visually on-‐screen and audibly to capture your think-‐aloud thoughts, in order to assist our ability to analyze your performance. Please rest assured that all video and audio we capture is held in strict confidence and will not be shared beyond our group. Before we begin the study, I need you to please sign this consent form indicating that you understand everything I’ve told you and reaffirm your agreement to participate. Additionally, I can answer any questions you have at this time.
Scanario & Tasks: Qlovi Scenario & Tasks: Scenario:
You are Ms. Patricia Lopez, a high-‐school literature teacher at Skyline High School in Ann Arbor. You generally teach tenth graders, who are about fifteen to sixteen years old. Your school has recently begun using Qlovi to deliver books electronically to students and you’d like to incorporate this system into your lesson plans.
Task #1
Locate and add a book to the library.
1-‐A:
Locate and add a book to the library. 1-‐b You need to assign a book to your students to read over the next two weeks. Based on the fact that you teach tenth graders, find one that you think is appropriate for your students’ reading level.
1-‐c:
You’ve found a book that you think is appropriate. Move it from the store to your personal library and ensure that it has been properly added.
Success:
User identifies a book either in the ‘+15 Years Old’ or ‘1200L – 1800L’ categories within the store and successfully adds it to his or her library. User must confirm the book has been added by going to the library to ensure it is accounted for.
Task #2
Create a module for a class.
2-‐A:
You need to create a module for your English 3-‐1 Class so that you can assign these readings. Name it after your book and add a short description.
Success:
The user successfully creates a module in the English 3-‐1 class and adds both an appropriate title and description.
Task #3
Modify and assign a module.
3-‐A:
Now that you’ve created the module, you need to modify it. Navigate to the module and add the book you just added to your library.
3-‐B:
You also need to add questions so you can assess whether your students will actually understand the reading. Add one quiz with two questions, one being in the form of ‘true or false’ and the second being an ‘open ended’ question. You can make the questions and answers to be anything you want. Don’t worry about adding common core standards.
Scanario & Tasks: Qlovi 3-‐C:
Now that your module is ready, you can release it. Assign it to the entire class so they can do the reading and answer the questions.
Success:
One module is created in English 3-‐1. This module has one assigned reading (The book from Task #1) and two questions, one of which is a ‘true or false’ question and the second being a ‘fill in the blank’ question.
Task #4
Separate students within a class into two groups.
4-‐A:
You notice that three of your students, Maria Gates, Carlos Smith, and Ricardo Rodriguez, in your English 3-‐1 class performed better on the assigned reading than their peers. Create a group within the class called ‘Advanced’ and assign them to it.
4-‐B:
The remaining students in your class didn’t perform as well. To keep the class organized, create a second group called ‘Regular’ and assign all remaining students to it.
Success:
There are two groups within the class, one named ‘Advanced’ with Maria, Carlos, and Ricardo in it, and the second named ‘Regular’ will all remaining students.
Task #5
Find a specific book in the Store and add it to an existing module.
5-‐A:
You’ve decided that you need find a second book for this module. One of your fellow teachers suggested that ‘The Raven’ [IF THEY ADDED ‘The Raven’ IN TASK #1, CHANGE THIS TO ‘The Bells’] by Edgar Allen Poe would be a good addition. Find that book and add it to the module as a required reading.
Success:
‘The Raven’ or ‘The Bells’ has been successfully located in the Qlovi Store and added to the module that was created in Task #2.
4/18/13
Pre Questionnaire
Pre Questionnaire These questions are designed to get your initial impression of Qlovi. * Required Please fill in your age *
Do you have teaching experience? * Yes No
The system looks like easy to use * Strongly disagree Disagree Neutral Agree Strongly agree
How would you rate your computer skills? * 1
2
3
4
5
Unskilled
Proficient
When I open the application, I immediately understood the function of each menu item. Strongly disagree Disagree Neutral Agree Strongly agree
How much prior experience you have with content or customer relationship management systems (CRM)? (i.e. Microsoft Dynamic, Zhoo, Sales Force, SAP, Interactive Excel Sheet) * 1 Unexperienced
2
3
4
5 Proficient
Submit
Never submit passwords through Google Forms. https://docs.google.com/a/umich.edu/spreadsheet/viewform?formkey=dFAydW13NGJ0Smt6UWNwSnhJSlFfeVE6MQ#gid=0
1/2
4/18/13
Post Questionnaire
Post Questionnaire Please rate the following statements. If the question is not applicable for you, please leave questions blank. * Required Qlovi would be useful in teaching 1
2
3
4
5
Strongly Disagree
Strongly Agree
Qlovi does everything that I would expect it to do. 1
2
3
4
5
Strongly Disagree
Strongly Agree
I found the system easy to use 1
2
3
4
5
Strongly Disagree
Strongly Agree
I found it easy to create a module 1
2
3
4
5
Strongly Disagree
Strongly Agree
I found it easy to assign a book to a module 1
2
3
4
5
Strongly Disagree
Strongly Agree
I found it easy to add questions in a module 1
2
3
4
5
Strongly Disagree
Strongly Agree
I understood the difference between "Library" and "Bookstore" in Qlovi 1 Strongly Disagree
2
3
4
5 Strongly Agree
Qlovi made easy to recover from errors I made https://docs.google.com/a/umich.edu/spreadsheet/viewform?formkey=dHk3Wlp5b0JOUGt0ajQwZlpTT2Z6SlE6MQ#gid=0
1/2
4/18/13
Post Questionnaire
1
2
3
4
5
Strongly Disagree
Strongly Agree
The aesthetics and design of Qlovi looks appealing * 1
2
3
4
5
Strongly Disagree
Strongly Disagree
I would recommend Qlovi to teachers 1
2
Strongly Disagree
3
4
5 Strongly Agree
Submit
Never submit passwords through Google Forms. Powered by Google Docs Report Abuse Terms of Service Additional Terms
https://docs.google.com/a/umich.edu/spreadsheet/viewform?formkey=dHk3Wlp5b0JOUGt0ajQwZlpTT2Z6SlE6MQ#gid=0
2/2
4/18/13
Edit form - [ Pre Questionnaire ] - Google Docs
5
responses
Summary See complete responses Please fill in your age 28 60 55 33 20
Do you have teaching experience? Yes
4
80%
No
1
20%
The system looks like easy to use Strongly disagree
0
0%
Disagree
0
0%
Neutral
0
0%
Agree
4
80%
Strongly agree
1
20%
How would you rate your computer skills? 1 Unskilled
0
0%
2
0
0%
3
0
0%
4
4
80%
5 Proficient
1
20%
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0ArGg6yYP1QoQdFAydW13NGJ0Smt6UWNwSnhJSlFfeVE&gridId=0#chart
1/3
4/18/13
Edit form - [ Pre Questionnaire ] - Google Docs
Unskilled
Proficient
When I open the application, I immediately understood the function of each menu item. Strongly disagree
0
0%
Disagree
0
0%
Neutral
2
40%
Agree
2
40%
Strongly agree
1
20%
How much prior experience you have with content or customer relationship management systems (CRM)? (i.e. Microsoft Dynamic, Zhoo, Sales Force, SAP, Interactive Excel Sheet) 1 Unexperienced 1
Unexperienced
20%
2
2
40%
3
2
40%
4
0
0%
5 Proficient
0
0%
Proficient
Number of daily responses
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0ArGg6yYP1QoQdFAydW13NGJ0Smt6UWNwSnhJSlFfeVE&gridId=0#chart
2/3
4/18/13
Edit form - [ Pre Questionnaire ] - Google Docs
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0ArGg6yYP1QoQdFAydW13NGJ0Smt6UWNwSnhJSlFfeVE&gridId=0#chart
3/3
4/18/13
Edit form - [ Post Questionnaire ] - Google Docs
5
responses
Summary See complete responses Qlovi would be useful in teaching 1 Strongly Disagree
0
0%
2
0
0%
3
2
40%
4
2
40%
5 Strongly Agree
1
20%
1 Strongly Disagree
0
0%
2
1
20%
3
1
20%
4
1
20%
5 Strongly Agree
2
40%
1 Strongly Disagree
0
0%
2
0
0%
3
2
40%
4
2
40%
5 Strongly Agree
1
20%
Strongly DisagreeStrongly Agree
Qlovi does everything that I would expect it to do.
Strongly DisagreeStrongly Agree
I found the system easy to use
Strongly DisagreeStrongly Agree
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0Ao1QVD52cwiIdHk3Wlp5b0JOUGt0ajQwZlpTT2Z6SlE&gridId=0#chart
1/4
4/18/13
Edit form - [ Post Questionnaire ] - Google Docs
I found it easy to create a module 1 Strongly Disagree
1
20%
2
0
0%
3
1
20%
4
1
20%
5 Strongly Agree
2
40%
1 Strongly Disagree
0
0%
2
1
20%
3
2
40%
4
0
0%
5 Strongly Agree
2
40%
1 Strongly Disagree
1
20%
2
1
20%
3
0
0%
4
1
20%
5 Strongly Agree
2
40%
2
40%
2
0
0%
3
2
40%
4
1
20%
5 Strongly Agree
0
0%
Strongly DisagreeStrongly Agree
I found it easy to assign a book to a module
Strongly DisagreeStrongly Agree
I found it easy to add questions in a module
Strongly DisagreeStrongly Agree
I understood the difference between "Library" and "Bookstore" in Qlovi 1 Strongly Disagree
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0Ao1QVD52cwiIdHk3Wlp5b0JOUGt0ajQwZlpTT2Z6SlE&gridId=0#chart
2/4
4/18/13
Edit form - [ Post Questionnaire ] - Google Docs
Strongly DisagreeStrongly Agree
Qlovi made easy to recover from errors I made 1 Strongly Disagree
0
0%
2
0
0%
3
1
20%
4
4
80%
5 Strongly Agree
0
0%
1 Strongly Disagree
0
0%
2
1
20%
3
2
40%
4
1
20%
5 Strongly Disagree
1
20%
1 Strongly Disagree
0
0%
2
0
0%
3
3
60%
4
1
20%
5 Strongly Agree
1
20%
Strongly DisagreeStrongly Agree
The aesthetics and design of Qlovi looks appealing
Strongly DisagreeStrongly Disagree
I would recommend Qlovi to teachers
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0Ao1QVD52cwiIdHk3Wlp5b0JOUGt0ajQwZlpTT2Z6SlE&gridId=0#chart
3/4
4/18/13
Edit form - [ Post Questionnaire ] - Google Docs
Strongly DisagreeStrongly Agree
Number of daily responses
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0Ao1QVD52cwiIdHk3Wlp5b0JOUGt0ajQwZlpTT2Z6SlE&gridId=0#chart
4/4
Note taking document User# :
Scenario & Tasks: Scenario: You are Ms. Patricia Lopez, a high-‐school literature teacher at Skyline High School in Ann Arbor. You generally teach tenth graders, who are about fifteen to sixteen years old. Your school has recently begun using Qlovi to deliver books electronically to students and you’d like to incorporate this system into your lesson plans. -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Pre-‐Test Notes:
Task #1
Locate and add a book to the library. 1-‐A: Locate and add a book to the library. 1-‐B You need to assign a book to your students to read over the next two weeks. Based on the fact that you teach tenth graders, find one that you think is appropriate for your students’ reading level. 1-‐C: You’ve found a book that you think is appropriate. Move it from the store to your personal library and ensure that it has been properly added. Success: User identifies a book either in the ‘+15 Years Old’ or ‘1200L – 1800L’ categories within the store and successfully adds it to his or her library. User must confirm the book has been added by going to the library to ensure it is accounted for.
Notes:
Note taking document Task #2 Create a module for a class. 2-‐A: You need to create a module for your English 3-‐1 Class so that you can assign these readings. Name it after your book and add a short description. Success: The user successfully creates a module in the English 3-‐1 class and adds both an appropriate title and description.
Notes: Task #3
Modify and assign a module.
3-‐A: Now that you’ve created the module, you need to modify it. Navigate to the module and add the book you just added to your library. 3-‐B: You also need to add questions so you can assess whether your students will actually understand the reading. Add one quiz with two questions, one being in the form of ‘true or false’ and the second being a ‘fill in the blank’. You can make the questions and answers to be anything you want. Don’t worry about adding common core standards. 3-‐C: Now that your module is ready, you can release it. Assign it to the entire class so they can do the reading and answer the questions. Success: One module is created in English 3-‐1. This module has one assigned reading (The book from Task #1) and two questions, one of which is a ‘true or false’ question and the second being a ‘fill in the blank’ question. Notes:
Task #4 Separate students within a class into two groups. 4-‐A: You notice that three of your students, Carlos Smith, Maria Gates, Ricardo Rodriguez in your English 3-‐1 class performed better on the assigned reading than their peers. Create a group within the class called ‘Advanced’ and assign them to it 4-‐B: The remaining students in your class didn’t perform as well. To keep the class organized, create a second group called ‘Regular’ and assign all remaining students to it. Success: There are two groups within the class, one named ‘Advanced’ with Gerado, Carlos, and Ricardo in it, and the second named ‘Regular’ will all remaining students.
Note taking document Notes:
Task #5 Find a specific book in the Store and add it to an existing module. 5-‐A: You’ve decided that you need find a second book for this module. One of your fellow teachers suggested that ‘The Raven’ [IF THEY ADDED ‘The Raven’ IN TASK #1, CHANGE THIS TO ‘The Bells’] by Edgar Allen Poe would be a good addition. Find that book and add it to the module as a required reading. Success: ‘The Raven’ or ‘The Bells’ has been successfully located in the Qlovi Store and added to the module that was created in Task #2. Notes:
Debrief / Post-Test
Notes:
Note Card for testers: Qlovi
Qlovi Login Information Account: Password:
lopez password123
Task 2: Module Name: Lesson 1 Module Description: Introduction of American History
Task 3: Assessment Detail: 1.True/False Question: American civil war ended in 1863? 2.Open Ended Question: Why was Toni Morrison’s novel, Song of Solomon, banned in many school districts in the country?
Task 4: Advance Group students: 1.Carlos Smith 2.Maria Gates 3.Ricardo Rodriguez Regular Group Students: Put the remain students in the regular group
Task 5: Book Name: Author:
The Raven Edgar Allen Poe
Debrief Plan: Qlovi For evaluators: What are your general impressions of Qlovi? [If the user did something unexpected] We noticed that during [task x] you [did something unexpected that we want to probe]. Would you care to explain why you did this? [If the user exhibits distress or has significant difficulty completing a task] I noticed you seemed to run into some difficulties during [task x], would you care to explain this? What features in Qlovi would you find most useful? Why? Would you like to use Qlovi if you were Mrs. Lopez? Why? [If post-test questionnaire answers don’t match observed behaviors] Looking over your answers to the questionnaire we noticed you said [answers that don’t match up with reactions] even though it seemed like [describe user’s reaction]. Could you explain why you answered this way? [If the user gives a strong negative response to in the questionnaire] In your questionnaire you indicated you were unhappy with [ x ]. can you tell me more about what bothered you? [If you are uncertain where the user missed something important] When you were doing [task x], do you remember seeing [allude to the area this important detail is located] Do you remember [what it was/said/did]? And lastly, before we let you go, we would like to ask if there are any last issues you would like to bring up to us.
For experimenters and observers: How did the session go? Were there any problems with the test? the debriefing session? Which part of tasks did the evaluators finish smoothly? In which part of tasks did they mostly get stuck? What are evaluators’ behaviors when he/she got stuck in [step x of task y]? What are major problems of Qlovi to prevent users finishing tasks? Why those problems occur? How to improve the system to solve those problems? How would you rank each issue’s priority?
Consent to Participate in a Research Study Research Title: Usability Testing and Evaluation on Qlovi 3.0 Principal Investigators: Chad Kijewski, Chin-Jui Chen, Hsiao-Chih Lin, Mohamed Gulaid, and SungJin Nam: Master Students, School of Information, University of Michigan Faculty Advisor: Mark Newman, Assistant Professor, School of Information, University of Michigan
Involvement Description If you agree to be part of the research study, you will be asked to perform several tasks on Qlovi, a e-reader system for k-12 students. After the tasks are completed, you will be given a survey and a short debrief about your experience using the Qlovi System. Throughout the duration of the study we will be recording your thoughts and actions with Camtasia and notes for future reference. Your participation is voluntary and you may stop the study at anytime. Your responses will be kept confidential. Benefits Although you may not directly benefit from being in this study, others may benefit because the results of this study will be used to improve instant messaging applications. Risks and discomforts This study poses no risk to your privacy and confidentiality. Further, the risk of physical harm is negligent and no different from any risk associated with ordinary use of a personal computer. Confidentiality You will not be identified in any reports on this study. Records will be kept confidential to the extent provided by federal, state, and local law. The data collected during this study will be kept on password-protected computers in the researchers’ locked offices and on devices securely stored in the researchers’ offices. The data collected and all data forms used will not contain any information that could be used to identify you. Instead, the forms and retained database will only contain a code number, linked to the participant’s identity by a key. This key will be kept separately from the data and will only be accessible by the research staff. Compensation You’ll be eligible in a 25$ Amazon gift card raffle. Consent I, __________________, have read the above statements and agree to participate in this
usability study and have my study session recorded.
___________________________ Signature
___________________________ Date
Environment Preparation Qlovi: Remove all groups in English 3-1: Class -> Group -> Select each Group -> Delete Group Remove all modules Module -> Delete each Module Remove all books in Library Library -> hover to book cover -> delete Laptop: Clear the cache in browser. You can use the one that you don’t use frequently. Setup and test microphone.
Introduction Script Introduction Script: Good [Morning/Afternoon/Evening] and thank you for agreeing to participate in this study. We’ve enlisted your help to conduct an evaluation of Qlovi, which is an electronic reading system aimed to cater to the needs of both students and teachers in primary schools. In order to accomplish this I will be guiding you through a scenario and asking you to accomplish five tasks within the system. Your performance will help us identify where Qlovi excels or needs improvement. Please work through the tasks to the best of your ability. Also, indicate to me when you feel that you have accomplished the task or do not want to continue so we can move on to the next. If you’d like, I can give you a few minutes to orient yourself to the system at this time. Also, I have a questionnaire that I would like you to fill out as well.
[If desired, the user orients him or herself to Qlovi for about two minutes] [User fills out the pre-‐test questionnaire]
I want to stress that this is an evaluation of the system and not your performance. There is no ‘right’ or ‘wrong’; observing any difficulties you experience while carrying out these tasks are just as valuable as your ability to accomplish them, if not more. Our team and school are not affiliated with Qlovi or its development team in any way, so please feel free to share your feelings about the system throughout the study. Your candid opinions are valuable and will not hurt anyone’s feelings. Please ‘think aloud’ while you are working on these tasks. By ‘think aloud’, I mean audibly communicate what you are thinking; what you are doing, why you are doing it, and what you plan on doing next. This will allow us to better understand where there may be disconnects between the system and the user. While you are completing these tasks, please pretend that I am not here. You may ask clarifying questions as I assign tasks throughout the study if you don’t understand something, but I will not be able to answer any questions while you are working on a specific task. If you don’t know how to do something, you don’t know how to do something. That’s valuable information we’re after. Also, I would like to make you aware that your performance will be recorded, both visually on-‐screen and audibly to capture your think-‐aloud thoughts, in order to assist our ability to analyze your performance. Please rest assured that all video and audio we capture is held in strict confidence and will not be shared beyond our group. Before we begin the study, I need you to please sign this consent form indicating that you understand everything I’ve told you and reaffirm your agreement to participate. Additionally, I can answer any questions you have at this time.
Scanario & Tasks Scenario & Tasks: Scenario:
You are Ms. Patricia Lopez, a high-‐school literature teacher at Skyline High School in Ann Arbor. You generally teach tenth graders, who are about fifteen to sixteen years old. Your school has recently begun using Qlovi to deliver books electronically to students and you’d like to incorporate this system into your lesson plans.
Task #1
Locate and add a book to the library.
1-‐A:
Locate and add a book to the library. 1-‐b You need to assign a book to your students to read over the next two weeks. Based on the fact that you teach tenth graders, find one that you think is appropriate for your students’ reading level.
1-‐c:
You’ve found a book that you think is appropriate. Move it from the store to your personal library and ensure that it has been properly added.
Success:
User identifies a book either in the ‘+15 Years Old’ or ‘1200L – 1800L’ categories within the store and successfully adds it to his or her library. User must confirm the book has been added by going to the library to ensure it is accounted for.
Task #2
Create a module for a class.
2-‐A:
You need to create a module for your English 3-‐1 Class so that you can assign these readings. Name it after your book and add a short description.
Success:
The user successfully creates a module in the English 3-‐1 class and adds both an appropriate title and description.
Task #3
Modify and assign a module.
3-‐A:
Now that you’ve created the module, you need to modify it. Navigate to the module and add the book you just added to your library.
3-‐B:
You also need to add questions so you can assess whether your students will actually understand the reading. Add one quiz with two questions, one being in the form of ‘true or false’ and the second being an ‘open ended’ question. You can make the questions and answers to be anything you want. Don’t worry about adding common core standards.
Scanario & Tasks 3-‐C:
Now that your module is ready, you can release it. Assign it to the entire class so they can do the reading and answer the questions.
Success:
One module is created in English 3-‐1. This module has one assigned reading (The book from Task #1) and two questions, one of which is a ‘true or false’ question and the second being a ‘fill in the blank’ question.
Task #4
Separate students within a class into two groups.
4-‐A:
You notice that three of your students, Maria Gates, Carlos Smith, and Ricardo Rodriguez, in your English 3-‐1 class performed better on the assigned reading than their peers. Create a group within the class called ‘Advanced’ and assign them to it.
4-‐B:
The remaining students in your class didn’t perform as well. To keep the class organized, create a second group called ‘Regular’ and assign all remaining students to it.
Success:
There are two groups within the class, one named ‘Advanced’ with Maria, Carlos, and Ricardo in it, and the second named ‘Regular’ will all remaining students.
Task #5
Find a specific book in the Store and add it to an existing module.
5-‐A:
You’ve decided that you need find a second book for this module. One of your fellow teachers suggested that ‘The Raven’ [IF THEY ADDED ‘The Raven’ IN TASK #1, CHANGE THIS TO ‘The Bells’] by Edgar Allen Poe would be a good addition. Find that book and add it to the module as a required reading.
Success:
‘The Raven’ or ‘The Bells’ has been successfully located in the Qlovi Store and added to the module that was created in Task #2.
4/18/13
Pre Questionnaire
Pre Questionnaire These questions are designed to get your initial impression of Qlovi. * Required Please fill in your age *
Do you have teaching experience? * Yes No
The system looks like easy to use * Strongly disagree Disagree Neutral Agree Strongly agree
How would you rate your computer skills? * 1
2
3
4
5
Unskilled
Proficient
When I open the application, I immediately understood the function of each menu item. Strongly disagree Disagree Neutral Agree Strongly agree
How much prior experience you have with content or customer relationship management systems (CRM)? (i.e. Microsoft Dynamic, Zhoo, Sales Force, SAP, Interactive Excel Sheet) * 1 Unexperienced
2
3
4
5 Proficient
Submit
Never submit passwords through Google Forms. https://docs.google.com/a/umich.edu/spreadsheet/viewform?formkey=dFAydW13NGJ0Smt6UWNwSnhJSlFfeVE6MQ#gid=0
1/2
4/18/13
Post Questionnaire
Post Questionnaire Please rate the following statements. If the question is not applicable for you, please leave questions blank. * Required Qlovi would be useful in teaching 1
2
3
4
5
Strongly Disagree
Strongly Agree
Qlovi does everything that I would expect it to do. 1
2
3
4
5
Strongly Disagree
Strongly Agree
I found the system easy to use 1
2
3
4
5
Strongly Disagree
Strongly Agree
I found it easy to create a module 1
2
3
4
5
Strongly Disagree
Strongly Agree
I found it easy to assign a book to a module 1
2
3
4
5
Strongly Disagree
Strongly Agree
I found it easy to add questions in a module 1
2
3
4
5
Strongly Disagree
Strongly Agree
I understood the difference between "Library" and "Bookstore" in Qlovi 1 Strongly Disagree
2
3
4
5 Strongly Agree
Qlovi made easy to recover from errors I made https://docs.google.com/a/umich.edu/spreadsheet/viewform?formkey=dHk3Wlp5b0JOUGt0ajQwZlpTT2Z6SlE6MQ#gid=0
1/2
4/18/13
Post Questionnaire
1
2
3
4
5
Strongly Disagree
Strongly Agree
The aesthetics and design of Qlovi looks appealing * 1
2
3
4
5
Strongly Disagree
Strongly Disagree
I would recommend Qlovi to teachers 1
2
Strongly Disagree
3
4
5 Strongly Agree
Submit
Never submit passwords through Google Forms. Powered by Google Docs Report Abuse Terms of Service Additional Terms
https://docs.google.com/a/umich.edu/spreadsheet/viewform?formkey=dHk3Wlp5b0JOUGt0ajQwZlpTT2Z6SlE6MQ#gid=0
2/2
4/18/13
Edit form - [ Pre Questionnaire ] - Google Docs
5
responses
Summary See complete responses Please fill in your age 28 60 55 33 20
Do you have teaching experience? Yes
4
80%
No
1
20%
The system looks like easy to use Strongly disagree
0
0%
Disagree
0
0%
Neutral
0
0%
Agree
4
80%
Strongly agree
1
20%
How would you rate your computer skills? 1 Unskilled
0
0%
2
0
0%
3
0
0%
4
4
80%
5 Proficient
1
20%
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0ArGg6yYP1QoQdFAydW13NGJ0Smt6UWNwSnhJSlFfeVE&gridId=0#chart
1/3
4/18/13
Edit form - [ Pre Questionnaire ] - Google Docs
Unskilled
Proficient
When I open the application, I immediately understood the function of each menu item. Strongly disagree
0
0%
Disagree
0
0%
Neutral
2
40%
Agree
2
40%
Strongly agree
1
20%
How much prior experience you have with content or customer relationship management systems (CRM)? (i.e. Microsoft Dynamic, Zhoo, Sales Force, SAP, Interactive Excel Sheet) 1 Unexperienced 1
Unexperienced
20%
2
2
40%
3
2
40%
4
0
0%
5 Proficient
0
0%
Proficient
Number of daily responses
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0ArGg6yYP1QoQdFAydW13NGJ0Smt6UWNwSnhJSlFfeVE&gridId=0#chart
2/3
4/18/13
Edit form - [ Pre Questionnaire ] - Google Docs
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0ArGg6yYP1QoQdFAydW13NGJ0Smt6UWNwSnhJSlFfeVE&gridId=0#chart
3/3
4/18/13
Edit form - [ Post Questionnaire ] - Google Docs
5
responses
Summary See complete responses Qlovi would be useful in teaching 1 Strongly Disagree
0
0%
2
0
0%
3
2
40%
4
2
40%
5 Strongly Agree
1
20%
1 Strongly Disagree
0
0%
2
1
20%
3
1
20%
4
1
20%
5 Strongly Agree
2
40%
1 Strongly Disagree
0
0%
2
0
0%
3
2
40%
4
2
40%
5 Strongly Agree
1
20%
Strongly DisagreeStrongly Agree
Qlovi does everything that I would expect it to do.
Strongly DisagreeStrongly Agree
I found the system easy to use
Strongly DisagreeStrongly Agree
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0Ao1QVD52cwiIdHk3Wlp5b0JOUGt0ajQwZlpTT2Z6SlE&gridId=0#chart
1/4
4/18/13
Edit form - [ Post Questionnaire ] - Google Docs
I found it easy to create a module 1 Strongly Disagree
1
20%
2
0
0%
3
1
20%
4
1
20%
5 Strongly Agree
2
40%
1 Strongly Disagree
0
0%
2
1
20%
3
2
40%
4
0
0%
5 Strongly Agree
2
40%
1 Strongly Disagree
1
20%
2
1
20%
3
0
0%
4
1
20%
5 Strongly Agree
2
40%
2
40%
2
0
0%
3
2
40%
4
1
20%
5 Strongly Agree
0
0%
Strongly DisagreeStrongly Agree
I found it easy to assign a book to a module
Strongly DisagreeStrongly Agree
I found it easy to add questions in a module
Strongly DisagreeStrongly Agree
I understood the difference between "Library" and "Bookstore" in Qlovi 1 Strongly Disagree
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0Ao1QVD52cwiIdHk3Wlp5b0JOUGt0ajQwZlpTT2Z6SlE&gridId=0#chart
2/4
4/18/13
Edit form - [ Post Questionnaire ] - Google Docs
Strongly DisagreeStrongly Agree
Qlovi made easy to recover from errors I made 1 Strongly Disagree
0
0%
2
0
0%
3
1
20%
4
4
80%
5 Strongly Agree
0
0%
1 Strongly Disagree
0
0%
2
1
20%
3
2
40%
4
1
20%
5 Strongly Disagree
1
20%
1 Strongly Disagree
0
0%
2
0
0%
3
3
60%
4
1
20%
5 Strongly Agree
1
20%
Strongly DisagreeStrongly Agree
The aesthetics and design of Qlovi looks appealing
Strongly DisagreeStrongly Disagree
I would recommend Qlovi to teachers
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0Ao1QVD52cwiIdHk3Wlp5b0JOUGt0ajQwZlpTT2Z6SlE&gridId=0#chart
3/4
4/18/13
Edit form - [ Post Questionnaire ] - Google Docs
Strongly DisagreeStrongly Agree
Number of daily responses
https://docs.google.com/a/umich.edu/spreadsheet/gform?key=0Ao1QVD52cwiIdHk3Wlp5b0JOUGt0ajQwZlpTT2Z6SlE&gridId=0#chart
4/4
Note taking document User# :
Scenario & Tasks: Scenario: You are Ms. Patricia Lopez, a high-‐school literature teacher at Skyline High School in Ann Arbor. You generally teach tenth graders, who are about fifteen to sixteen years old. Your school has recently begun using Qlovi to deliver books electronically to students and you’d like to incorporate this system into your lesson plans. -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Pre-‐Test Notes:
Task #1
Locate and add a book to the library. 1-‐A: Locate and add a book to the library. 1-‐B You need to assign a book to your students to read over the next two weeks. Based on the fact that you teach tenth graders, find one that you think is appropriate for your students’ reading level. 1-‐C: You’ve found a book that you think is appropriate. Move it from the store to your personal library and ensure that it has been properly added. Success: User identifies a book either in the ‘+15 Years Old’ or ‘1200L – 1800L’ categories within the store and successfully adds it to his or her library. User must confirm the book has been added by going to the library to ensure it is accounted for.
Notes:
Note taking document Task #2 Create a module for a class. 2-‐A: You need to create a module for your English 3-‐1 Class so that you can assign these readings. Name it after your book and add a short description. Success: The user successfully creates a module in the English 3-‐1 class and adds both an appropriate title and description.
Notes: Task #3
Modify and assign a module.
3-‐A: Now that you’ve created the module, you need to modify it. Navigate to the module and add the book you just added to your library. 3-‐B: You also need to add questions so you can assess whether your students will actually understand the reading. Add one quiz with two questions, one being in the form of ‘true or false’ and the second being a ‘fill in the blank’. You can make the questions and answers to be anything you want. Don’t worry about adding common core standards. 3-‐C: Now that your module is ready, you can release it. Assign it to the entire class so they can do the reading and answer the questions. Success: One module is created in English 3-‐1. This module has one assigned reading (The book from Task #1) and two questions, one of which is a ‘true or false’ question and the second being a ‘fill in the blank’ question. Notes:
Task #4 Separate students within a class into two groups. 4-‐A: You notice that three of your students, Carlos Smith, Maria Gates, Ricardo Rodriguez in your English 3-‐1 class performed better on the assigned reading than their peers. Create a group within the class called ‘Advanced’ and assign them to it 4-‐B: The remaining students in your class didn’t perform as well. To keep the class organized, create a second group called ‘Regular’ and assign all remaining students to it. Success: There are two groups within the class, one named ‘Advanced’ with Gerado, Carlos, and Ricardo in it, and the second named ‘Regular’ will all remaining students.
Note taking document Notes:
Task #5 Find a specific book in the Store and add it to an existing module. 5-‐A: You’ve decided that you need find a second book for this module. One of your fellow teachers suggested that ‘The Raven’ [IF THEY ADDED ‘The Raven’ IN TASK #1, CHANGE THIS TO ‘The Bells’] by Edgar Allen Poe would be a good addition. Find that book and add it to the module as a required reading. Success: ‘The Raven’ or ‘The Bells’ has been successfully located in the Qlovi Store and added to the module that was created in Task #2. Notes:
Debrief / Post-Test
Notes:
Note Card for testers
Qlovi Login Information Account: Password:
lopez password123
Task 2: Module Name: Lesson 1 Module Description: Introduction of American History
Task 3: Assessment Detail: 1.True/False Question: American civil war ended in 1863? 2.Open Ended Question: Why was Toni Morrison’s novel, Song of Solomon, banned in many school districts in the country?
Task 4: Advance Group students: 1.Carlos Smith 2.Maria Gates 3.Ricardo Rodriguez Regular Group Students: Put the remain students in the regular group
Task 5: Book Name: Author:
The Raven Edgar Allen Poe
Consent to Participate in a Research Study Research Title: Usability Testing and Evaluation on Qlovi 3.0 Principal Investigators: Chad Kijewski, Chin-Jui Chen, Hsiao-Chih Lin, Mohamed Gulaid, and SungJin Nam: Master Students, School of Information, University of Michigan Faculty Advisor: Mark Newman, Assistant Professor, School of Information, University of Michigan
Involvement Description If you agree to be part of the research study, you will be asked to perform several tasks on Qlovi, a e-reader system for k-12 students. After the tasks are completed, you will be given a survey and a short debrief about your experience using the Qlovi System. Throughout the duration of the study we will be recording your thoughts and actions with Camtasia and notes for future reference. Your participation is voluntary and you may stop the study at anytime. Your responses will be kept confidential. Benefits Although you may not directly benefit from being in this study, others may benefit because the results of this study will be used to improve instant messaging applications. Risks and discomforts This study poses no risk to your privacy and confidentiality. Further, the risk of physical harm is negligent and no different from any risk associated with ordinary use of a personal computer. Confidentiality You will not be identified in any reports on this study. Records will be kept confidential to the extent provided by federal, state, and local law. The data collected during this study will be kept on password-protected computers in the researchers’ locked offices and on devices securely stored in the researchers’ offices. The data collected and all data forms used will not contain any information that could be used to identify you. Instead, the forms and retained database will only contain a code number, linked to the participant’s identity by a key. This key will be kept separately from the data and will only be accessible by the research staff. Compensation You’ll be eligible in a 25$ Amazon gift card raffle. Consent I, __________________, have read the above statements and agree to participate in this
usability study and have my study session recorded.
___________________________ Signature
___________________________ Date
Debriefing Plan For evaluators: What are your general impressions of Qlovi? [If the user did something unexpected] We noticed that during [task x] you [did something unexpected that we want to probe]. Would you care to explain why you did this? [If the user exhibits distress or has significant difficulty completing a task] I noticed you seemed to run into some difficulties during [task x], would you care to explain this? What features in Qlovi would you find most useful? Why? Would you like to use Qlovi if you were Mrs. Lopez? Why? [If post-test questionnaire answers don’t match observed behaviors] Looking over your answers to the questionnaire we noticed you said [answers that don’t match up with reactions] even though it seemed like [describe user’s reaction]. Could you explain why you answered this way? [If the user gives a strong negative response to in the questionnaire] In your questionnaire you indicated you were unhappy with [ x ]. can you tell me more about what bothered you? [If you are uncertain where the user missed something important] When you were doing [task x], do you remember seeing [allude to the area this important detail is located] Do you remember [what it was/said/did]? And lastly, before we let you go, we would like to ask if there are any last issues you would like to bring up to us.
For experimenters and observers: How did the session go? Were there any problems with the test? the debriefing session? Which part of tasks did the evaluators finish smoothly? In which part of tasks did they mostly get stuck? What are evaluators’ behaviors when he/she got stuck in [step x of task y]? What are major problems of Qlovi to prevent users finishing tasks? Why those problems occur? How to improve the system to solve those problems? How would you rank each issue’s priority?
Environment Preparation Qlovi: Remove all groups in English 3-1: Class -> Group -> Select each Group -> Delete Group Remove all modules Module -> Delete each Module Remove all books in Library Library -> hover to book cover -> delete Laptop: Clear the cache in browser. You can use the one that you don’t use frequently. Setup and test microphone.
