ONLINE PIZZA ORDERING PROJECT EVALUATION REPORT AND SUMMARY
Submitted by Team C:
Suroor Fatima This document contains the usability study analysis and project evaluation report for the Tasty Pizzeria’s Online Pizza Ordering system.
Rodrigo Sardinas Gabriel Stone Christiana Unaeze
Date Submitted: Thursday, December 06, 2012
Page 1 of 56
1 Contents 1.
Detailed plan for evaluating the Online Pizza Ordering System ..................................................... 3 1.1
Introduction ............................................................................................................................ 3
1.2
Purpose ................................................................................................................................... 4
1.3
Scope and Limitations ............................................................................................................. 4
1.4
Team Roles .............................................................................................................................. 4
1.5
Method ................................................................................................................................... 5
1.6
Participants ............................................................................................................................. 5
1.7
Materials ................................................................................................................................. 6
1.8
Desktop Interface Usability test plan ...................................................................................... 6
1.8.1
Procedure ........................................................................................................................ 6
1.8.2
Participants of the Desktop interface ............................................................................. 6
1.8.3
User Tasks for the Desktop interface .............................................................................. 7
1.9
1.9.1
Procedure ........................................................................................................................ 7
1.9.2
Participants of the Website interface ............................................................................. 8
1.9.3
Website Interface User Tasks.......................................................................................... 8
1.10
2
3
Website Interface Usability test plan...................................................................................... 7
Mobile Interface Usability test plan........................................................................................ 8
1.10.1
Procedure ........................................................................................................................ 8
1.10.2
Participants of the Mobile interface ............................................................................... 9
1.10.3
Mobile Interface User Tasks............................................................................................ 9
Signed consent forms.................................................................................................................... 10 2.1
Desktop interface usability testing: Signed consent forms by participants ......................... 10
2.2
Website interface usability testing: Signed consent forms by participants ......................... 15
2.3
Mobile interface Signed consent forms by participants ....................................................... 20
Usability testing ............................................................................................................................ 25 3.1
Usability Testing Procedure .................................................................................................. 25
3.1.1
Set-up Sheet .................................................................................................................. 25
3.1.2
Checklist: ....................................................................................................................... 25
3.2
Preamble ............................................................................................................................... 25
3.3
Task List ................................................................................................................................. 26
3.3.1
Desktop Task List ........................................................................................................... 26
3.3.2
Website Task List........................................................................................................... 27
3.3.3
Mobile Task List............................................................................................................. 27
Page 2 of 56
4
3.4
Qualitative Data .................................................................................................................... 27
3.5
Desktop Interface Questionnaire .......................................................................................... 28
3.6
Website Interface Questionnaire.......................................................................................... 33
3.7
Mobile Interface Questionnaire ............................................................................................ 38
Analysis of the test results ............................................................................................................ 41 4.1
4.1.1
Observations ................................................................................................................. 42
4.1.2
Questionnaire results .................................................................................................... 43
4.2
6
7
Website Interface Analysis.................................................................................................... 43
4.2.1
Observations ................................................................................................................. 44
4.2.2
Questionnaire results .................................................................................................... 45
4.3
5
Desktop Interface Analysis .................................................................................................... 41
Mobile Interface Analysis...................................................................................................... 45
4.3.1
Observations ................................................................................................................. 46
4.3.2
Questionnaire Results ................................................................................................... 47
Summary of the usability study analysis ....................................................................................... 47 5.1
Desktop Interface summary .................................................................................................. 47
5.2
Website Interface study summary ........................................................................................ 48
5.3
Mobile Interface summary.................................................................................................... 48
Conclusions ................................................................................................................................... 48 6.1
Desktop Interface conclusions .............................................................................................. 48
6.2
Website Interface Usability Study conclusions ..................................................................... 48
6.3
Mobile Interface Usability Study conclusions ....................................................................... 49
Recommendations ........................................................................................................................ 49 7.1
Desktop Interface recommendations ................................................................................... 49
7.2
Website Interface recommendations ................................................................................... 52
7.3
Mobile Interface recommendations ..................................................................................... 54
Page 3 of 56
Online Pizza Ordering System Usability Study and Project Evaluation Report 1. Detailed plan for evaluating the Online Pizza Ordering System 1.1 Introduction This is a report of the usability test plan and study conducted to analyze the user interface of the Tasty Pizzeria’s pilot online pizza ordering system. Tasty pizzeria is a small business Pizza parlor located in a single location with the potential to expand operations to other locations. They are very popular locally for their selection of pizzas and sides but are losing business to local competition due to lack of online ordering convenience offered to customers. Tasty Pizzeria has launched three interfaces as part of their online ordering systems:
Figure 1: Desktop interface
Figure 2: Website Interface
Figure 3: Mobile Interface
1. Desktop interface: Order from the desktop application for repeat customers 2. Website interface: Order from the website 3. Mobile interface: Order from the mobile website for phone users
Page 4 of 56
1.2 Purpose The International Organization for Standardization or ISO's definition of usability is the "effectiveness, efficiency and satisfaction with which a specified set of users can achieve a specified set of tasks in particular environments. The purpose of this usability study was to assess the design of the desktop, website and mobile user interfaces of the online Pizza ordering system for effectiveness, efficiency and user satisfaction. The focus of the study was to observe a typical user’s experience in interacting with the visual and functional design of the three interfaces individually and determine the strengths and weakness of the different usability attributes that contribute to the overall pizza ordering experience.
1.3 Scope and Limitations The scope of the study is limited to the basic user scenarios which have been currently implemented: Interface
Desktop
Website
Mobile
Add Pizza to order
Supported
Supported
Supported
Add non Pizza items to order Modify selections in the current order Place order
Supported
Supported
Supported
Supported
Not Supported
Not Supported
Supported
Supported
Not Supported
Customize Pizza
Supported
Not Supported
Not Supported
View Order history
Supported
Not Supported
Not Supported
1.4 Team Roles The interview of participants using questionnaires, conducting the usability study using active observation of test subjects, evaluation of the results for the assigned interface was performed by as follows: Interface Desktop interface usability study, Final evaluation report compile , organize, edits and submission Website interface usability study Mobile interface usability study
Team member Suroor Fatima Rodrigo Sardinas Gabriel Stone,, Rodrigo Sardinas and Christiana Unaeze
Page 5 of 56
1.5 Method Five participants were recruited for conducting the usability study for each of the three interfaces: Desktop, website and mobile. Hence a total of 15 participants were invited to take part in this usability study. The method of observation is both direct as well as using remote desktop. The participants were given representative sample scenarios for making an online order using one of the three assigned interfaces. The participants were not instructed on how to perform the tasks but just briefed on the background of the project to support Tasty pizzeria’s new pilot project for online ordering of pizzas. Participants were shown the opening screen for the each of the three pizza ordering interfaces (see Figures 1-3). The focus of the study was to observe the interaction of the users with the user design of the supported features as a starting point to improve user interface design for effectiveness, efficiency and overall customer satisfaction. Features not currently implemented for website and mobile interfaces cannot be tested by users and are not included in their sample scenarios at this point.
1.6 Participants The users are chosen with different demographics as follows: Audience Profile: Current Profession example: Student, Employee, Stay home mom etc Age Range: 1. 2. 3. 4.
18-25 26-39 40-59 60+
Computer usage in hours per week: 1. 0 to 10 hrs per wk. 2. 11 to 25 hrs per wk. 3. 26+ hrs per wk. Gender: 1. Choices: Male, Female
Page 6 of 56
1.7 Materials The following materials are required for the usability testing:
Usability Testing Participation Consent forms Task lists Observation sheets Questionnaires Desktop with the opening screen of the assigned user interface for testing Remote Desktop Screen sharing using Skype for remote subjects Timers
1.8 Desktop Interface Usability test plan 1.8.1 Procedure In order to evaluate the Desktop interface implementation, the usability study participants representing the real world users are provided with Tasty pizzeria’s desktop buddy to place the online pizza order. The testing is performed in a controlled environment with a mix of users being observed directly and using a remote screen share session. At no point are users prompted on how to perform the tasks , they are free to use their judgment on how to interact with the UI elements of the interface to accomplish the given task. Observations are to be recorded using both team members as well as a non-team member observer to provide a non-biased view on the effectiveness of the system. After each task, users rated the perceived task difficulty on a 5-point Liker scale, (1=very difficult, 5=very easy) and completed a satisfaction survey pertaining to their interaction with that particular app. After completing all tasks, users can “play” with the system and give their feedback based upon their free form interaction. 1.8.2
User User 1
Participants of the Desktop interface
Age range 18-25
Gender
Profession
FEMALE
User 2
26-39
MALE
User 3
26-39
MALE
User 4
26-39
MALE
User 5
40-59
FEMALE
STUDENT TECHNICAL WRITER SUPPORT ENGINEER PROGRAM MANAGER GRAND MOTHER
Figure 4: User Demographics of the Desktop Interface Usability Study
Computer experience 26+ 26+ 26+ 26+ 11 to 25
Page 7 of 56 1.8.3 User Tasks for the Desktop interface The usability subjects were given access to the opening screen of desktop application Tasty Pizzeria desktop buddy and asked complete a series of tasks:
Login using existing account (test account given) Add a pizza to their order Add a cheese pizza and customize toppings Delete an item from the current order Add a non-pizza item to their order Order a variety of items using a $30 budget. Choose an item from previous orders and add to current cart. Confirm selections and place order
1.9 Website Interface Usability test plan In order to evaluate the Website Interface Implementation, the plan is to observe people using the Tasty Pizzeria website interface in a monitored environment. There will be one observant in the room, as well as another person (not related to the project) observing the screen remotely. The purpose of using the observant not related to the project is to provide a non-biased opinion on the success and effectiveness of the participant’s interaction with the interface. 1.9.1 Procedure The website interface user will be asked to complete various tasks without any direct instructions or guidance from the team or from any observant. The purpose is to see how easily the user can accomplish the tasks and infer what needs to be done without any direction. This will help determine whether or not the interface was intuitive for the purpose of ordering the pizza online. After the user has completed the tasks, they will be allowed several minutes to interact freely by simply browsing over the site pages in the manner they determine best in order to perform free form testing scenarios not listed in their assigned tasks. This will help understand what features most users will expect to see in a pizza ordering site, to determine whether the user interface elements were effective and met the expectations of a typical user who would want to order a pizza online. Following the users interaction with the interface, the user will be given a questionnaire where they will be asked specific questions about the tasks they had to perform (how easy/difficult it was, etc..) and they will also be allowed to comment on additional features that they felt the site lacked or features that they felt should have been implemented differently (from a user’s point of view – not the developer’s).
Page 8 of 56 1.9.2 User
Participants of the Website interface Age Gender Profession
User 1
40-59
FEMALE
User 2 User 3 User 4 User 5
18-25 26-39 26-39 18-25
MALE MALE MALE MALE
SALES STUDENT STUDENT STUDENT STUDENT
Computer experience 11 TO 25 26+ 26+ 26+ 26+
Figure 5: User Demographics of the Website Interface Usability Study
1.9.3
Website Interface User Tasks Create a new pizza order without logging into any account Login to an existing account Login anonymously without creating an account Create and place a new pizza order Complete an order and checkout Make payment using mock credit card number Provide optional additional comments and suggestion regarding the website site
1.10 Mobile Interface Usability test plan In order to evaluate the Mobile Interface Implementation, the plan is to observe people using the mobile website in a monitored environment using a remote observant. The remote observant will be monitoring the session via the Windows Remote Assistance tool. 1.10.1 Procedure The mobile interface users are provided with sample representative tasks for ordering a pizza using a mobile site. The typical users would use a phone in order to place the orders. How ever since the mobile website is not live at this point, the users were provided with a test environment simulating a mobile website for smartphones on a desktop computer. Due to time and resource constraints the mobile interface for the online Pizza project is still under construction and not live on a web server that a user can visit from a smartphone. The subject will be asked to complete various tasks without any assistance from the development team. The purpose of the testing is to see how well the user can perform the various tasks without any direction. Upon completion of assigned tasks, the subject will be asked to complete a questionnaire regarding their experience. The questionnaire asks questions regarding various features of the site including account creation, placing orders, etc…, and also their thoughts regarding the usefulness and effectiveness of various features.
Page 9 of 56 1.10.2 Participants of the Mobile interface User Age Gender Profession User 1 User 2 User 3 User 4 User 5
18-25 18-25 26-39 18-25 18-25
MALE MALE MALE MALE MALE
STUDENT STUDENT STUDENT STUDENT STUDENT
Computer experience 26+ 26+ 26+ 26+ 26+
Figure 6: User Demographics of the Mobile Interface Usability Study
1.10.3 Mobile Interface User Tasks Create a new pizza order without logging into any account Login to an existing account Login anonymously without creating an account Create and place a new pizza order Complete an order and checkout Provide optional additional comments and suggestion regarding the mobile site
Page 10 of 56
2 Signed consent forms 2.1 Desktop interface usability testing: Signed consent forms by participants
Page 11 of 56
Page 12 of 56
Page 13 of 56
Page 14 of 56
Page 15 of 56
2.2 Website interface usability testing: Signed consent forms by participants
Page 16 of 56
Page 17 of 56
Page 18 of 56
Page 19 of 56
Page 20 of 56
2.3 Mobile interface Signed consent forms by participants
Page 21 of 56
Page 22 of 56
Page 23 of 56
Page 24 of 56
Page 25 of 56
3 Usability testing 3.1 Usability Testing Procedure 3.1.1 Set-up Sheet The Usability Test team will use the Set-up sheet to ensure that the testing room is set-up properly before the users arrive. For remote test subject it is ensured that the machine hosting the screen share is ready with the opening screen of the assigned test interface. It will include a checklist of necessary material and equipment for the test, as well as the preamble that will be given to the participants. 3.1.2 Checklist: Virtual machines and/or PCs with Windows OS and the Application installed for the specific assigned user interface. Ensure that the application is running properly on each Virtual machine or PC.
Task list Pencils for each user. Consent form for each user. Questionnaires for each user.
3.2 Preamble The primary usability test facilitator will speak first to greet participants and thank them for participating in the usability testing. She/he will explain the value of usability testing and introduce themselves and the non- team observer. She/he will also indicate that the interface is in its early stages, and that it is possible for anything to change at this point in time. Narrative: “We would like you to run through a series of tasks for us, to see how well the product does its job. Running through the complete set of tasks should take approximately 1 hour. Please remember that you are not being tested; in fact, you are doing the testing. Consequently, if you can't figure out how to complete a task, it is not your fault, but the fact that user interface is not "user-friendly." The purpose of the testing is to find out these very things. You have in front of you a PC with an online Pizza ordering application running, a task list, questionnaire and pencil. If you are missing any of these items, please let us know so that we can provide you with them. Please read the instructions for each task carefully, and follow the instructions step-by-step. The online Pizza ordering application is designed to help customers order Pizzas from the convenience of their home, office or on the go using desktop interface, website interface and mobile interfaces. You will have one specific type of interface assigned to you to play with and test if it works for ordering menu items from Tasty pizzeria’s online menu selection.
Page 26 of 56 The online Pizza ordering application that you will see is a prototype that is in the very early stages of development. The purpose of this testing is to get feedback from users before the design is too far along. Also, you will notice that each of you will be provided with test accounts for the assigned interfaces. The test accounts will not be provided to actual customers. Also the website interface takes a mock credit card for payment. There is no actual transfer of money involved. We would you to follow through the tasks listed on the task sheet you have been given. Please answer all questions that are listed, and we encourage you to make as many comments in the "Notes" section of the task sheet as possible. We also encourage you to make comments, think aloud as you work through the tasks, as this will help us uncover problems with the product. While you are completing the tasks, a member of the usability test team will observe how you complete the tasks, and record these observations on an Observation sheet. Again, the observer wants to see how you complete the task - there is not a right and wrong way. If you have questions or require assistance, please ask the observer. However, we would like you to attempt to complete each task without any help, and ask questions related to the tasks only if you are really stuck or the application has crashed.”
3.3 Task List Users will be given a task list that outlines the tasks they must perform during the test. Each task in the test will have an associated question and evaluation scale. A five-point scale, with appropriate labels, will be used for the usability testing. The users will circle the appropriate number on the scale, as well as write any comments they may have in the "Notes" section. 3.3.1 Desktop Task List Instructions Task 1: Login using the Tasty Pizzeria desktop buddy with the following credentials: Username: dhaskin Password: pizza Task 2: Place an order for any items of your choice from the menu selection under a budget of $30.00. Task 3: Start a new order and order a plain cheese pizza with extra sauce, light cheese and topped with bell peppers and chicken. Complete the order and view the order in order history. Task 4: Place an order with any item of your choice and pick any item from your previous orders. Complete this order with home delivery schedule on the same day at 8pm. Task 5: Start a new order and select 2 sides and 1 dessert along with a custom pizza. Logout and log back in again and resume order from where you left off. Before confirming the order, go back and remove one of the sides, then proceed to confirm this order for carryout with a coupon code of 10PERCENTOFF Task 6: Use the interface as you seem appropriate to order items of your choice.
Page 27 of 56
3.3.2 Website Task List Instructions Task 1: Create a new user account using the user name and password of your choice and login into your account. Task 2: Start an order for any items of your choice from the menu selection under a budget of $30.00. Make sure you add at least one pizza and one non-pizza item example: side or drink. Task 3: Confirm the selected menu items and place the order. Task 4: Make payment using the mock credit card number. Task 5: Logout from the currently logged in user account. Task 6: Use the interface as you seem appropriate to order items of your choice. 3.3.3 Mobile Task List Instructions Task 1: Login using an existing account. Task 2: Start an order and add one item from the menu selection to the order for carryout. Task 3: Confirm the selected menu items and place the order. Task 4: Use the interface as you seem appropriate to order items of your choice.
3.4 Qualitative Data Users will be encouraged to talk aloud during the testing. They should vocalize their problem-solving strategies, give their points-of-view, and make whatever comments they like throughout the session. Members of the Usability Test Team will take notes throughout the session in order to collect this qualitative data. The members of the Usability Test Team will record their observations on Observation sheets during the testing.
Page 28 of 56
3.5 Desktop Interface Questionnaire Please provide the following information so we can further develop this application to be more usable and efficient. Tasty Pizzeria Desktop Interface Questionnaire Response Notes Task 1
Login using the Tasty Pizzeria desktop buddy with the following credentials: Username: dhaskin Password: pizza
Q1
Did you complete this task successfully?
Yes
Yes No
Q2
Rate your experience using this task. Circle one choice:
Very Easy
1 - Very Difficult 2- Difficult 3- Average 4- Easy 5- Very Easy
Q3
Were you able to land on the My Account tab?
Yes
Yes No
Q4
Were you able to navigate easily from the My Account tab to other tabs?
Yes
Yes No
Q5
Task 2
Q6
Additional comments on the process of Login using preexisting account. Liked the big green button for Login and the same button becoming Logout upon successful login. It makes it easy to find it instead of searching for it in a different location. Place an order for any items of your choice from the menu selection under a budget of $30.00.
Did you complete this task successfully?
Yes
Yes No
Q7
Rate your experience using this task. Circle one choice: 1 - Very Difficult 2- Difficult 3- Average 4- Easy
Easy
Page 29 of 56 5- Very Easy
Q8
Were you able to navigate to the menu selection tab easily?
Yes
Yes No
Q9
Were you able to navigate easily from the menu selection tab to other tabs?
Yes
Yes No
Q10
Additional comments on the process of Place order process using preexisting account. Easy to add menu items using both text entry and numeric control. Wish for drag and drop items into the cart. Love the cart showing current items but make the cart clickable to allow checkout or view cart directly from the cart icon.
Task 3
Start a new order and order a plain cheese pizza with extra sauce, light cheese and topped with bell peppers and chicken. Complete the order and view the order in order history.
Q11
Did you complete this task successfully?
Yes
Yes No
Q12
Rate your experience using this task. Circle one choice:
Easy
1 - Very Difficult 2- Difficult 3- Average 4- Easy 5- Very Easy
Q13
Were you able to navigate to the customize pizza selection tab easily?
Yes
Yes No
Q14
Were you able to navigate easily from the customize menu selection tab to other tabs? Yes No
Yes
Page 30 of 56 Q15
Task 4
Q16
Additional comments on the process of customizing the pizza order process using preexisting account.
On first landing on the Customize pizza tab, it takes a good few seconds before the pizzas are available in the dropdown box to allow customization. Confusing to the user if the system is still working or there is no pizza to customize. There should be some kind of indication for the user to wait. Place an order with any item of your choice and pick any item from your previous orders. Complete this order with home delivery schedule on the same day at 8pm. Did you complete this task successfully?
Yes
Yes No
Q17
Rate your experience using this task. Circle one choice:
Average
1 - Very Difficult 2- Difficult 3- Average 4- Easy 5- Very Easy
Q18
Were you able to navigate to the Place orders selection tab easily?
Yes
Yes No
Q19
Were you able to navigate easily from the Place orders menu selection tab to other tabs?
Yes
Yes No
Q20
Were you able to navigate to the Items recently ordered selection tab easily?
Yes
Yes No
Q21
Were you able to navigate easily from the Items recently selection tab to other tabs?
Yes
Yes No
Q22
Did the order get scheduled for the current date with 8pm delivery? Yes No
Yes
Page 31 of 56 Q23
Additional comments on the process of customizing the pizza.
Drag and drop of toppings into the pizza would make it easier. I like the customization options and the graphics.
Task 5
Start a new order and select 2 sides and 1 dessert along with a custom pizza. Logout and log back in again and resume order from where you left off. Before confirming the order, go back and remove one of the sides, then proceed to confirm this order for carryout with a coupon code of 10PERCENTOFF.
Q24
Did you complete this task successfully?
Yes
Yes No
Q25
Rate your experience using this task. Circle one choice:
Easy
1 - Very Difficult 2- Difficult 3- Average 4- Easy 5- Very Easy
Q26
Were you able to add non pizza items easily?
Yes
Yes No
Q27
Were you able to modify quantities of the items added to the cart easily?
Yes
Yes No
Q28
Were you able to view the cart contents consistently across tabs?
Yes
Yes No
Q29
Were you able to logout and resume the order successfully?
Yes
Yes No
Q30
Did the order get scheduled for the current date carryout?
Yes
Yes No
Q31
Were you able to apply the coupon code successfully to get the discount on your order? Yes No
Yes
Page 32 of 56
Q32
Additional comments on the process of adding variety items in the current order and placing order process using preexisting account.
Modifying cart contents from the Place order tab is currently not allowed. It makes it easier to add/remove items if this feature is made available. Task 6
Q33
Use the interface as you seem appropriate to order items of your choice. Played with adding desserts, sides, drinks etc and picking items from previous orders. Did you complete this task successfully?
Yes
Yes No
Q34
Rate your experience using this task. Circle one choice: 1 - Very Difficult 2- Difficult 3- Average 4- Easy 5- Very Easy
Q35
Additional comments Previous ordered items show up repeated for each order. It should show the items only once. Liked the image preview of the previous ordered items.
Easy
Page 33 of 56
3.6 Website Interface Questionnaire Below is one of the Questionnaires with user’s responses filled in: Tasty Pizzeria Web Interface Questionnaire
1
2
USER ACCOUNT CREATION Where you able to successfully create a User Account? Yes No If not, please explain why.
yes
3
Was the process difficult? Yes No
no
4
Were you able to easily start and complete the process without instruction from anyone? Yes No Additional comments on the process of creating a user account.
yes
5
LOGGING IN
6
7
8
Were you able to successfully log in to your user account? Yes No If not, please explain why.
yes
Was the process difficult? Yes No
no
Page 34 of 56 9
Were you able to easily start and complete the process without instruction from anyone? Yes No 10 Additional comments on the process of logging in.
yes
LOGGING OUT
11 Were you able to successfully log out of your account? Yes No 12 If not, please explain why.
yes
13 Was the process difficult? Yes No
no
14 Were you able to easily start and complete the process without instruction from anyone? Yes No 15 Additional comments on the process of logging out.
yes
PLACING AN ORDER AFTER LOGGING IN
16 Were you able to successfully create and place an order after logging in? Yes No
yes
Page 35 of 56 17 If not, please explain why.
18 Was the process difficult? Yes No
no
19 Were you able to easily start and complete the process without instruction from anyone? Yes No 20 Additional comments on the process of placing an order after logging in.
yes
PLACING AN ORDER WITHOUT LOGGING IN
21 Were you able to successfully create and place an order without first logging in? Yes No 22 If not, please explain why.
no
23 Was the process difficult? Yes No
no
24 Were you able to easily start and complete the process without instruction from anyone? Yes No 25 Additional comments on the process of placing an order without first logging in.
I could create the order but had to log in to complete it.
Page 36 of 56 ADDING / EDITING / MANIPULATING PAYMENT OPTIONS
26 Were you able to successfully select a payment method when completing your order? Yes No 27 If not, please explain why.
yes
28 Were you able to successfully delete a payment method? Yes No 29 If not, please explain why.
yes
30 Were you able to successfully edit a payment method? Yes No 31 If not, please explain why.
yes
32 Were you able to successfully add a payment method? Yes No 33 If not, please explain why.
yes
34 Were you able to easily start and complete all of these processes without instruction from anyone? Yes No 35 Additional comments on the process of manipulating payment information.
yes
Adding or editing a new payment method was a little bit difficult at first. The way we had to format it is not normal.
Page 37 of 56 COMPLETING AN ORDER
36 Were you able to successfully complete an order?
yes
37 If not, please explain why.
38 During the process of creating and completing your order, was your order information always displayed, and was it always displayed correctly?
yes
39 Did you, at any time, find it difficult to locate your order information?
no
40 Were you notified of the complete order details upon completing your order?
yes
41 Additional comments on the process of completing an order.
It was very simple
ADDITIONAL COMMENTS ON INTERFACE USABILITY AND EXPECTATIONS
42 Please include any additional comments you have about the time you were allowed to freely browse the interface without instructions or tasks to complete.
I felt like the interface did not offer as many features as most websites do. Several of the pages were blank and had no content.
Page 38 of 56
3.7 Mobile Interface Questionnaire Below is one of the Questionnaires with user’s responses filled in: Tasty Pizzeria Mobile Interface Questionnaire LOGGING IN WITH AN ACCOUNT
1
2
3
4
Were you able to successfully login to your user account? If not, why? Yes No Was the process difficult? If so, why?
yes
Were you able to easily start and complete the login process without instruction from anyone? If not, why? Yes No Additional comments on the process of logging in with an account.
yes
no
CREATING AN ORDER AFTER LOGGING IN
5
6
7
Were you able to successfully create an order after logging in? If not, why? Yes No Was the process difficult? If so, why? Yes No
yes
Were you able to easily start and complete the process without instruction from anyone? If not, why? Yes No
yes
no
Page 39 of 56 8
Additional comments on the process of creating an order after logging in.
LOGGING IN ANONYMOUSLY WITHOUT AN ACCOUNT
9
10
11
12
Were you able to enter in delivery information and login without an account? If not, why? Yes No Was the process difficult? If so, why? Yes No
yes
Were you able to easily start and complete the login process without instruction from anyone? If not, why? Yes No Additional comments on the process of logging in anonymously without an account.
yes
no
CREATING AN ORDER WITHOUT LOGGING IN
13
14
15
Were you able to successfully create and place an order without first logging in? If not, why? Yes No Was the process difficult? If so, why? Yes No
no. I could create the order but could not complete it. The page gave me some kind of error.
Were you able to easily start and complete the process without instruction from anyone? If not, why?
no. I could create the order but could not complete it. The page gave me some kind of error.
yes. I was able to create my order but could not complete it.
Page 40 of 56 16
Additional comments on the process of creating an order without first logging in.
same problem that I get when I log in. cannot complete it.
CHECKING OUT
17
18
19
20
Were you able to successfully complete an order and check out, receiving confirmation? If not, why? Yes No At the end of the checkout process, was your order information displayed, and displayed correctly? Yes No Did you, at any time, find it difficult to locate your order information? Yes No Additional comments on the process of checking out.
ADDITIONAL COMMENTS ON INTERFACE USABILITY AND EXPECTATIONS
21
Please include any additional comments you have about the time you were allowed to freely browse the mobile interface without instructions or tasks to complete.
no. I don't know.
no.
yes. It was not there.
Does not work.
Page 41 of 56
4 Analysis of the test results 4.1 Desktop Interface Analysis Interface: Desktop User
Time taken to add Pizza in Seconds
User 1 User 2 User 3 User 4 User 5 Total Time Taken by All Users Average
Time taken to add other menu selections in Seconds
12.1 5.9 15 20 19 10 76.1 15.22
Time taken to modify order in Seconds
Time taken to place order in Seconds
8 7.4 19.3 11.5 16.1
6.4 10.2 15.3 22 59.8
62.3
11.96
12.46
13 18 14.6 26 19.2
Time taken to Customize Pizza (NA if feature not implemente d) in Seconds 25 13 14.3 23.9 17.3
90.8
93.5
18.16
18.7
Overall Satisfaction rating (1-5) 1 least 5 most
4 5 4 5 5
4.6
Figure 7: Data collected from the usability study for time taken to perform tasks using desktop interface
Average Time Taken to Perform Tasks using Desktop Interface Time taken to Customize Pizza (NA if… Time taken to place order in Seconds Time taken to modify order in Seconds Average
Time taken to add other menu selections… Time taken to add Pizza in Seconds 0
5
10
Figure 8: Comparison of time taken for different tasks using the desktop interface
15
20
Page 42 of 56
Overall Satisfaction rating for Destktop interface (1-5) 1 least 5 most 6 5 4 3
Overall Satisfaction rating (15) 1 least 5 most
2 1 0 User 1
User 2
User 3
User 4
User 5
Figure 9: User satisfaction rating for the desktop interface
4.1.1 Observations It was observed that all of the users managed to handle all of the assigned tasks very easily. The users were very satisfied with the following:
Tabbed layout and wizard style navigation. Image previews of the menu items. Visual preview of the pizza along with number of slices graphic. Graphic icons for the toppings. Dynamic menu selection automatically updating the cart. Able to logout and not lose the cart contents if order was not placed. Easy empty cart and empty toppings buttons. Login and Logout button location being the same. Color scheme and graphics. Ordering a variety of menu items. Viewing the order total. Specifying delivery preference, scheduling preference and coupon code for discounts. Easy to use controls and intuitive organization and layout. Ability to see order history.
Their satisfaction level was low in the following areas:
Connection error was not conveyed properly when the network connection to the SQL Server was down. The user is not able to edit their home delivery address or provide some other address for the pizza delivery. The user is not able to add/edit/modify their order from the place order tab. The user is not able to interact with the cart icon and perform a checkout. Some users recommended drag and drop functionality for menu selections and toppings.
Page 43 of 56 They did not seem to have to search for which controls to use, it seemed as if all of the controls were placed in a manner that made them easy to find and use. The users found the experience to be very rich and needed a lot of time to test the various features and play with them. The time taken by the users for the various tasks is in the expected range and depends upon the nature of the order. The Toppings selection took more time than the menu selection as the choice was quite varied and the user carefully examined the choices. 4.1.2 Questionnaire results Based on the results from the questionnaire all of the users seemed to be able to perform all of the tasks easily. Users were happy with the desktop implementation and scored well on the satisfaction rating for the various tasks. Some users recommended that the previously ordered pizzas along with the toppings should be made available ready to use for the next order. Also some users asked for a feature to save the pizzas under favorites so that they do not have to make the topping selections repeatedly for their favorites.
4.2 Website Interface Analysis Interface: Website User
User 1 User 2 User 3 User 4 User 5 Total Time Taken by All Users Average
Time taken to add Pizza in Seconds
20 10 8 9 7 54
10.8
Time taken to add other menu selections in Seconds
2 2 2 2 2 10
2
Time taken to modify order in Seconds
NA NA NA NA NA
NA NA
Time taken to place order in Seconds
Overall Satisfaction rating (1-5) 1 least 5 most
88 75 67 69 61 360
Time taken to Customize Pizza (NA if feature not implemente d) in Seconds NA NA NA NA NA NA
72
NA
3.2
Figure 10: Data collected from the usability study for time taken to perform tasks using website interface
3 4 3 2 4
Page 44 of 56
Average Time Taken to Perform Tasks using Website Interface
Time taken to place order in… Time taken to add other menu… Average Time taken to add Pizza in… 0
20
40
60
80
Figure 11: Comparison of time taken for different tasks using the website interface
Overall Satisfaction rating for Website interface (1-5) 1 least 5 most 5 4 3 Overall Satisfaction rating (1-5) 1 least 5 most
2 1 0 User 1
User 2
User 3
User 4
User 5
Figure 12: User satisfaction rating for the website interface
4.2.1 Observations It was observed that all of the users managed to handle all of the assigned tasks very easily. They did not seem to have to search for which controls to use, it seemed as if all of the controls were placed in a manner that made them easy to find and use. I did notice that at the end, when the users were asked to browse for additional tasks that weren’t supplied, most of them felt like the site was very barren and that it did not really offer what they were expecting, based on previous experiences. The ease of use estimation might be skewed since all of the participants were from the computer science department, and as such, I would consider them to be advanced users. Therefore, we don’t really have any data as to how a novice user might find the site. We did not notice much of any delay when the users were executing their tasks. They seemed to be able to perform all of the tasks easily and in a reasonable amount of time.
Page 45 of 56 4.2.2 Questionnaire results Based on the results from the questionnaire all of the users seemed to be able to perform all of the tasks easily. Some of the users seemed to find a few aspects of the tasks a little tedious (manipulating payment methods), and one user mentioned that they weren’t happy with being forced to allow the company to keep a record of their credit card details. They later mentioned that if it were an actual website they would not use it for that reason alone. I believe that most users also found the website too minimal. Based on past experiences with similar sites they expected more functionality and felt that the website was incomplete.
4.3 Mobile Interface Analysis Interface: Mobile User
Time taken to add Pizza in Seconds
Time taken to add other menu selections in Seconds
User 1
6
5
User 2
4
4
User 3 User 4
5 5
4 5
User 5
4
4
24
22
4.8
4.4
Time taken to modify order in Seconds
NA NA NA NA NA Total Time Taken by All Users Average
Time taken to place order in Seconds
Time Overall taken to Satisfaction Customize rating (1-5) Pizza (NA if feature not implemen ted) in Seconds NOT NA 3 COMPLETED NOT NA 2 COMPLETED 168 NA 2 NOT NA 3 COMPLETED NOT NA 3 COMPLETED NOT NA COMPLETED
NA NA
NOT COMPLETED
NA
Figure 13: Data collected from the usability study for time taken to perform tasks using mobile interface
2.6
Page 46 of 56
Average Time Taken to Perform Tasks using Mobile Interface Time taken to add other menu selections in Seconds Average
Time taken to add Pizza in Seconds 4.2
4.4
4.6
4.8
Figure 14: Comparison of time taken for different tasks using the mobile interface
Overall Satisfaction rating for Mobile interface (1-5) 1 least 5 most rating (1-5) 4 3 2
Overall Satisfaction rating (1-5)
1 0 User 1
User 2
User 3
User 4
User 5
Figure 15: User satisfaction rating for the mobile interface
4.3.1 Observations We felt that all of the users managed to handle most of the assigned tasks very easily. They did not seem to have to search for which controls to use, it seemed as if all of the controls were placed in a manner that made them easy to find and use. The only difficulty any of the users experienced was when attempting to check out, since this feature is not completely implemented yet. Some actually did manage to complete the checkout process. Some users experienced stability issues when trying to checkout at times causing the website to crash. When the users were asked if they were satisfied with the application, most of them replied that for a mobile app / website, the app was sufficient (except for the bug of course, and the fact that even if it did work, there was no way to enter payment information and they would realistically not be able to actually place the order unless they had to pay upon arrival or by visiting the store) and that they would not expect much else from a mobile app.
Page 47 of 56 The ease of use estimation might be skewed since all of the participants were from the computer science department, and as such, we would consider them to be advanced users. Therefore this study may not be true representation of a varied sample user audience from the real world. 4.3.2 Questionnaire Results Based on the results from the questionnaire all of the users seemed to be able to perform most of the supported tasks easily. Some of the users seemed to be a bit confused since they were not able to actually view their order details and never really had an idea what they had actually ordered. However the study is useful in order to incorporate the feedback for a fully functional mobile website.
5 Summary of the usability study analysis 5.1 Desktop Interface summary This study evaluated the functional, visual, performance and subjective perceptions of five users recruited to perform online pizza ordering tasks using the Tasty pizzeria’s Desktop interface called Desktop buddy. The five users were selected with a varied age groups, gender and professional profile range. However most of them were very experienced with computer usage and typically engaged in 26+ hours per week computer usage. The five users were asked to perform a series of tasks without specifying the mechanics of how to perform the tasks. The users were able to successfully login, add menu selections of a variety of items, modify the cart contents, select from previously ordered items, place order and view order history. In addition users had the rich choice of customizing their pizzas for crust, sauce, cheese, toppings – both vegetarian and meats. The flow of the navigation was easy to understand and intuitive allowing users to access any tab from their current tab location or using the wizard style next button to move from one tab to other in a sequential manner. The users really liked the image preview, size preview and the number of slices shown visually for the size of the pizza selected. This took the guess work out of the user’s mind on how many slices the pizza will contain and how many people it can feed. Viewing the image preview for the drinks, sides, specialty pizzas and desserts makes the menu selection process informational and enjoyable. The users liked the cart icon but wished it was clickable and interactive providing an easy shortcut to check out the current cart or view the cart contents. A few subjects’ desired drag and drop facility for the menu selection but this may not be an efficient way to manage data entry for quick orders using the keyboard. The tab order of the controls is sequential allowing users to use the desktop interface without using the mouse and placing an order with a couple of quick taps. Overall the desktop implementation provides a full fidelity experience of enabling the user to make menu selections from the main menu and/or previously ordered items along with customizing and placing order using delivery preferences, scheduling preferences and discount coupons.
Page 48 of 56
5.2 Website Interface study summary This study evaluated the functional, visual, performance and subjective perceptions of five users recruited to perform online pizza ordering tasks using the Tasty pizzeria’s pilot website. The users performed a series of tasks while the observers took note of their impressions, perceived difficulty, eye motions, time taken to perform the tasks and overall satisfaction levels. Results showed that users reported the UI to be appealing for the website but were not satisfied by the lack the features expected in an online ordering website. User loved to see the menu beforehand and view the images of the items prior to ordering. However they also wanted to view their orders, add/edit and modify orders, provide coupons for discounts in order to make the online ordering a more efficient process than making the order via phone call. Since this was the first implementation of the prototype, there is potential to incorporate the feedback from this study and create a website which allows users to place orders easily from the website instead of making a call to place the order from Tasty pizzeria. Based on the results from the questionnaire all of the users seemed to be able to perform all of the tasks easily. Some of the users seemed to find a few aspects of the tasks a little tedious (manipulating payment methods), and one user mentioned that they weren’t happy with being forced to allow the company to keep a record of their credit card details. They later mentioned that if it were an actual website they would not use it for that reason alone.
5.3 Mobile Interface summary We decided to go strictly with monitored user testing as the primary method for evaluating our interface. Our users were asked to complete specific tasks and were monitored by an in-room monitor as well as a remote viewer (via Windows Remote Assistance) who was not part of the project and was able to offer an impartial opinion of the ease of the user’s experience. In addition, we offered the users a questionnaire to complete when they were finished, where they would have a chance to answer specific questions about their assigned tasks, as well as offer additional comments on issues we may not have foreseen when designing the questionnaire. Once their assigned tasks were complete, they were allowed to simply browse the site on their own, as if it were a live site, and offer additional insight into other features we may not have included or other thoughts they may have had that were not addressed by the specific tasks that we gave them. We felt that the combination of these methods offered the most thorough review that we could get from the users.
6 Conclusions 6.1 Desktop Interface conclusions 6.2 Website Interface Usability Study conclusions The primary goal of placing an online order is achieved using the provided website interface. The website is functional for the basic scenario of choosing menu items and placing an order. Users enjoyed being able to view the images of the products in the website menu and appreciated the quick checkout flow. In addition it allows users to create an account and make payment using their choice of stored credit cards. Although this is not commercial quality as it is in the development stages, the UI design has a great visual appeal with the color scheme and layout matching the user expectations for an online pizza ordering website. There are known stability issues as the project is
Page 49 of 56 in prototype level and is an excellent resource to gain usability feedback from users prior to full scale implementation. Given the time and resources, the recommendations made in this report for the website interface can enable the website users to enjoy a more efficient and effective user experience for ordering their pizza online from Tasty pizzeria.
6.3 Mobile Interface Usability Study conclusions In conclusion, we achieved our main goal of being able to offer a very simple and easy to use mobile interface for ordering pizza. Users can add a single item and place an order by specifying the delivery address for the order. Although the application is not yet complete, we think that we are at least heading in the right direction with the overall design of this part of the application. Although the initial round of testing was performed using the handheld HTC smart phone, the usability study was conducted using a simulated environment with the project individually set up for each station. The users were unable to access the mobile live site due to technical issues with the master page. The prototype is buggy and not complete in features at this time due to challenges with resources and time during the course of implementation of this interface. However there is significant learning made in the area of user interface design for mobile scenarios during the course of this project.
7 Recommendations 7.1 Desktop Interface recommendations Based upon the usability study the following recommendations are made:
Recommendation 1: Allow user to manage their account directly from the desktop interface.
Page 50 of 56
Recommendation 2: Allow anonymous user to complete checkout and create account automatically upon completing the order by prompting for user name and password.
Recommendation 3: Send email confirmation to users to allow them to see order details of their order placed.
Recommendation 4: Provide user with an interactive cart icon which shows the current cart contents upon clicking the icon and enables easy checkout.
Page 51 of 56
Recommendation 5: Allow user to add/modify/delete current order items from the Place order tab.
Recommendation 6: Show only distinct items from previous orders for pre-selection.
Page 52 of 56
Recommendation 7: Once the user is logged in the login panel should be visually dimmed.
Recommendation 8: Allow user to provide any delivery address other than the Home address.
7.2 Website Interface recommendations Based upon the usability study the following recommendations are made:
Recommendation 1: Provide user a cart for managing the current order selections.
Page 53 of 56
Recommendation 2: Send email confirmation to users to allow them to see order details of their order placed.
Recommendation 3: Allow customer to add/delete/modify cart contents.
Recommendation 4: Allow customization of the pizza to allow user to specify toppings etc. Recommendation 5: The menu items are very long and the user has to scroll through a very long page to see the menu items for selection.
Page 54 of 56
Recommendation 6: Instead of Finish button – rename it to Place order or a more suitable verb describing the action.
Recommendation 7: Fix the tedious long scrolling menu page with appropriate categories and organization and paging.
7.3 Mobile Interface recommendations Based upon the usability study the following recommendations:
Recommendation 1: Provide user a cart for managing the current order selections.
Recommendation 2: Allow anonymous user to complete checkout and create account automatically upon completing the order by prompting for user name and password.
Page 55 of 56
Recommendation 3: Send email confirmation to users to allow them to see order details of their order placed.
Recommendation 4: Allow customer to add/delete/modify cart contents.
Recommendation 5: Allow customization of the pizza to allow user to specify toppings etc. Recommendation 6: Allow user to select multiple menu items in a single order. Recommendation 7: Instead of Finalize order link – rename it to Place order or a more suitable verb describing the action.
Page 56 of 56
Recommendation 8: Keep the UI color scheme consistent with the main website and the desktop interface.
