Summary of SYDE 348 Course Project (Winter 2001)
Usability Testing of the UW Library Gateway Web Page

Prepared for: UW Library Community Needs Assessment Group
Prepared by: Prof. Carolyn MacGregor, Scott Anderson, and Lora Bruyn
Dept of Systems Design Engineering

June 4, 2001

Table of Contents

1. OBJECTIVES
   1.1Objectives of SyDe 348 (User-Centred Design) Course
   1.2 Objectives of UW Library Gateway Web Page Project
2. PROJECT REQUIREMENTS & CONSTRAINTS
   2.1 Requirements
   2.2 Project Constraints
3. USABILITY TESTING & EVALUATION METHODS
   3.1 Phase One

   3.1.1 Heuristic Evaluations
   3.1.2 Cognitive Walkthroughs
   3.1.3 Hierarchical Task Analysis
   3.1.4 Competitive Analysis

   3.2 Phase Two

   3.2.1 Card Sorting Technique
   3.2.2 Discount Usability

   3.3 Phase Three

   3.3.1 Keystroke-Level Analysis
   3.3.2 Lab-Based User Testing (Timed Trial and Subjective Evaluation)
4. EVALUATION OF THE UW LIBRARY GATEWAY WEB PAGE
5. COMMON SOLUTIONS FROM TEAM PROTOTYPES
6. RESULTS OF USABILITY TESTING
7. DESIGN RECOMMENDATIONS
8. PROJECT CONCLUSIONS

SUMMARY OF SYDE 348 COURSE PROJECT (WINTER 2001)
USABILITY TESTING OF THE UW LIBRARY GATEWAY WEB PAGE

1. OBJECTIVES

   1.1 Objectives of SyDe 348 (User-Centred Design) Course
   The overall objective of SyDe 348 is to expose upper year students to user-centred design (UCD) methods through lectures, hand-on class activities, and a term-long team project. Most students enter the course with some background in human factors engineering, but with no experience in usability testing. The students are assigned to work in multi-disciplined teams for the course project. The UW Library Community Needs Assessment Group (CNAG) provided the topic for this year’s course project.

   1.2 Objectives of UW Library Gateway Web Page Project
   The overall objective of the UW Library Gateway Web Page Project was for each team to conduct an initial usability evaluation of the current UW Library Gateway, and to redesign (and evaluate those redesigns) through a series of user-centred design techniques. All teams were to have a functional prototype working by the second last week of the term, so that all teams could participate in lab-based testing. Each team was assigned a member of CNAG who would serve as the team’s "client". Prof. MacGregor and her two teaching assistants served as project managers and project co-ordinators. Prof. MacGregor specified which UCD methods were to be used in each of the three phases of the project. Each team used a total of 8 different UCD methods to help with the development and evaluation of their UW Library Gateway designs.

2. PROJECT REQUIREMENTS & CONSTRAINTS

   Project requirements and constraints are presented from the perspective of the students, who served as designers and evaluators.

   2.1 Requirements
   The overall goal of the course project can be stated as an Interactive Systems Problem Statement:

   Design a University of Waterloo Library Gateway web page that is easy to use for library and resource-related activities by members of the UW community and public at large.

   The general requirements of the students were as follows:

   • Students were to evaluate the current Library Gateway page using methods covered in the course
   • Students were to produce at least three prototypes representing progressive levels of fidelity
   • Students had to have a functional prototype available for user testing in a lab-based setting by March 22, 2001 (x weeks from the start of the project)
   • Students were to prepare reports at the end of each phase of usability testing
   • Students were to submit a final version of the report in electronic format by the end of the exam period

   2.2 Project Constraints
   The general project constraints faced by the students were as follows:
   • Students were limited as to the time available for working on the project as most students were carrying a full course load and this was just one of 5 or 6 courses.
   • Most of the students had only limited or no previous experience with website design
   • Most of the students had only limited or no previous experience with formal usability evaluation methods
   • Most students were limited to the software available to them through their department accounts for creating the web pages

3. USABILITY TESTING & EVALUATION METHODS

   The project was divided into three phases; each with its specific UCD methods and design goals. In Phase One, students were instructed to assess the current UW Library Gateway design and to propose a design that could be tested using paper-based techniques. In Phase Two, the teams put their proposed designs on "8 ½ X 11" paper. These low-fidelity prototypes were tested before the teams decided upon the design that they would implement as a medium-fidelity prototype in HTML for Phase Three. Phase Three testing involved structured lab-based testing that included timed user trials and subjective evaluations of the medium-fidelity prototypes.

   3.1 Phase One (Current UW Library Gateway Design)
   In Phase One of the project, students were asked to evaluate the current gateway web page using four different techniques: heuristic evaluations, cognitive walkthroughs, hierarchical task analysis, and competitive analysis.

   3.1.1 Heuristic Evaluations.
   Heuristic evaluation is one of the most common methods of usability inspection. The evaluator reviews the product using a set of general principles or rules-of-thumb. The more experienced the evaluator, the more effective the technique. The students of SyDe 348 were introduced to two general sets of heuristics: Donald Norman’s Four Principles of Usable Design, and Jakob Nielsen’s Ten Usability Heuristics (See Appendix A). Most of the teams elected to use Nielsen’s heuristics.

   3.1.2 Cognitive Walkthroughs.
   Designers use cognitive walkthroughs to get a sense of what the user might experience in carrying out a task. Specific tasks or scenarios are chosen, and the designer steps through the task just as if she is the user. The teams were encouraged to have at least two or three of their members carry out cognitive walkthroughs using the usability questions posed by CNAG as the starting points for the user tasks.

   3.1.3 Hierarchical Task Analysis.
   Task Analysis is a common technique used to assess the efficiency of a task. A hierarchical task analysis (HTA) breaks a task down into its component steps so that the designer can determine whether steps can be integrated through a design change to save time, or whether steps need to be broken down further to simplify a task that is too complex for the user. Rather than carrying out HTAs on all possible tasks related to the Gateway page, teams were instructed to carry out an HTA on a representative task.

   3.1.4 Competitive Analysis.
   Most product designers evaluate the products of their competitors so as to get an idea of how their own design might surpass the competition. CNAG provided the teams with a list of library sites of other universities. The teams were asked to review a set of these sites and assess them for weaknesses (that should be avoided in the UW site), and for strengths (that might be incorporated into the new UW design).

   3.2 Phase Two (Low-Fidelity Prototype)
   Teams continued to revise their designs by testing their own low-fidelity (paper-based) prototype using two main methods: card sorting, and discount usability.
   3.2.1 Card Sorting Technique.
   In Phase One, all of the teams identified the keywords and major links on the current Gateway to pose challenges for users. Card sorting is a technique that allows designers to get a sense of how users intuitively group concepts together. In essence, it provides some insight into the user’s mental model of how a product should work. The basic steps applied to a web page design is to provide the user with stack of cards, each one representing an existing link or keyword associated with the current design. Blank cards are also provided should the user think of a link that they think should be included on the page. The user then sorts the cards into what he believes to be "natural" categories for the web page under review. Users may be instructed to sort into a specific number of categories (e.g. 2, 4, 6) or may be allowed to select the number of categories he thinks are needed.

   3.2.2 Discount Usability.
   Discount Usability is a method developed by Jakob Nielsen to allow companies unable to afford formal lab-based usability testing to conduct user testing that is effective and economical. Discount usability involves quick cycling of heuristic evaluations followed by design walkthroughs with 2 or 3 representative users. The designer(s) carries out a heuristic evaluation of the design and makes modifications based on that evaluation before carrying out design walkthroughs with 2 or 3 users. Design problems identified with the walkthroughs are then fixed prior to carrying out another cycle of heuristic evaluations followed by walkthroughs. Students were instructed to carry out at least two discount usability cycles with their low-fidelity prototype(s).

   3.3 Phase Three (Medium-Fidelity Prototype)
   At the end of Phase Two, each team was to propose the design that it would implement in HTML code for more formal usability testing. For the Phase Three testing all teams followed the testing protocol submitted to the Office of Research and Ethics. In order to determine, the target times for each trial (i.e. minimum time if the task was error free), each team carried out a keystroke level analysis for all of the test tasks.
   3.3.1 Keystroke-Level Analysis (KLA).
   One way of assessing the efficiency of computer tasks is to calculate how quickly an experienced user might be able to carry out the task without errors. This type of analysis can be used to compare one proposed design to another without actually involving users. A KLA can also help a design team to set benchmarks for experienced users, and against which they can compare times for novice users.

   3.3.2 Lab-Based User Testing (Timed Trials and Subjective Evaluation).
   An engineering computer lab was booked for a morning in the second last week of classes so that all of the students in the class could participate as test monitors and experience formal lab-based usability testing. Each student was required to recruit two participants for testing. Participants were asked to carry out a set of timed tasks using the team’s web page design, and to then answer questions pertaining to the usability of the web page they used.

4. EVALUATION OF THE CURRENT UW LIBRARY GATEWAY WEB PAGE

   Each team produced its own final report documenting findings and proposed solutions for each phase of the project. Summaries for each team can be found in Appendix B. Below is a list of the most common usability issues associated with the current design (as identified by the teams):

   • Words and phrasing used for keywords and major links are not intuitive
     • Examples:
       • TRELLIS does not quickly convey to the user that it is the catalogue
       • Find It and Get It do not intuitively convey their subsets of links
   • The large graphic of the Davis Centre windows is distracting and obscures the Text Version link
   • The navigation bar at the top of the page is often missed. It is not clear to all users that they represent buttons for "UW Home", "Contact/Email" or a quick link to the main catalogue
   • The mouse-over features and menu types are not used consistently. The result is extra steps or backtracking for the user.
   • Some users expected a direct search function on the main library page. Many of the better competitor sites include a direct search function.

5. COMMON SOLUTIONS FOR TEAM PROTOTYPES

   Each team decided how it was going to address the usability issues they identified in Phase One. Across the six teams, there were the following common solutions:

   • Removed main graphic (often included a graphic more representative of UW)
   • Reorganized and renamed main links (and sub-links)
   • Allowed for direct search on the Gateway
   • Provided quick links to anticipated main tasks
   • Made "Help" link more salient

   Team prototypes can be found in the team final reports.

6. RESULTS OF FORMAL USABILITY TESTING

   In order to identify the more effective and usable designs, we have compared the usability testing of the various team designs against that for the current UW Library design. Table 1 presents the results from the timed trials. Based on a comparison of average trial time, we can see that the design put forward by Team 4 generated fairly short trial times for all 10 tasks.

   Table 1: Results of Performance Testing (Average Trial Time in seconds)

   › = Fastest time › = second fastest time › = third fastest time

   		Team Number
   Tasks performed	Current	1	2	3	4	5	6
   Where would you go to find general help with a research topic in your department?	14 s	25 s	25 s	38 s	22 s	32 s	41 s
   How do you get an article from a journal that is available at WLU?	10 s	15 s	17 s	38 s	6 s	10 s	13 s
   Can Alumni borrow books from UW?	6 s	30 s	31 s	7 s	13 s	75 s	15 s
   What are the hours of the University Archives?	18 s	5 s	10 s	10 s	15 s	12 s	6 s
   What books are available at KPL?	22 s	5 s	10 s	45 s	16 s	23 s	12 s
   Does the Library have electronic dictionaries?	49 s	40 s	19 s	10 s	16 s	31 s	35 s
   Does the Library have a copy of Andrew Pyper’s Lost Girls?	8 s	8 s	7 s	10 s	7 s	45 s	27 s
   Where can you find instructions for connecting from home?	19 s	23 s	42 s	8 s	6 s	11 s	16 s
   Where would you go to renew books online?	10 s	8 s	12 s	7 s	33 s	5 s	16 s
   What is the contact info for the liaison librarian assigned to your home department?	18 s	24 s	27 s	45 s	11 s	20 s	31 s

   Table 2 presents the results from the subjective evaluations of the usability of the web page designs. Team 3’s design achieved consistently high usability ratings. Team 4 also achieved good ratings on 4 of the 5 categories (ratings for graphics were low because specific graphics were not included on the prototype tested).

   Table 2: Frequency that users described web page as "very user friendly"

   › = best score › = second best score › = third best score

   	Team Number
   	Current	1	2	3	4	5	6
   Key Words	3/5 (60%)	7/10 (70%)	4/7 (57%)	8/9 (89%)	9/11 (82%)	8/9 (89%)	5/10 (50%)
   Major Links	2/5 (40%)	8/10 (80%)	4/13 (31%)	9/9 (100%)	10/11 (91%)	8/9 (89%)	7/10 (70%)
   Graphics	3/5 (60%)	6/10 (60%)	11/13 (85%)	7/9 (78%)	7/11 (64%)	3/9 (33%)	3/10 (30%)
   Navigation Bars	3/5 (60%)	8/10 (80%)	8/13 (62%)	7/9 (78%)	5/11* (46%)	1/9* (11%)	6/10* (60%)
   Overall layout	3/5 (60%)	8/10 (80%)	10/13 (77%)	9/9 (100%)	9/11 (82%)	2/9 (22%)	6/10 (60%)

   * Prototypes of Teams 4,5, and 6 did not include graphics other than colour borders

7. DESIGN RECOMMENDATIONS FOR NEW UW LIBRARY GATEWAY

   Taking into consideration the results of the usability testing, we examined each of the six team designs in order to arrive a composite of the best usability features (See Figure 1). The most effective features from the team designs included:

   • A clear logo UW logo (top left corner)
   • A quick search feature (top left)
   • A section of quick links for more experienced users (left side of web page)
   • A simplified navigation bar that can be used on sub-layers and that includes a site index (even on the home page)
   • A clear link for the Text Version (top right corner)
   • The majority of the space devoted to main links and sublinks
   • A clear visual connection between a major link and its sublinks
   • Use of colours and fonts that facilitate reading and feedback (see http://www.useit.com for information pertaining to link designs)

   The wording for the major links and sub-links need to be reconsidered. We recommend that a more thorough card sorting exercise be carried out with a wider range of representative users in order to clarify the conceptual structure and phrasing for the major links.

   While not part of this project, many of the students noted the following usability issues that should be addressed at the sub-layers associated with the UW Library web site:

   • Work towards consistency between sub-layers.
     • A site template is needed.
   • Include feedback when the user leaves the UW Library web site.
     • Many complex and heavily linked sites are going with dialog boxes that inform user she is leaving the site, or that open up a separate active window for distinct sites.
   • Reduce the number of intermediary steps that take the user to more information instead of directly to the site.
     • Example: To view your patron information, selecting "Get it" then "View your Record" takes you to an information page that tells you that you can view your patron information through TRELLIS.
   • Avoid use of colour red as a highlight colour as it is troublesome for users who are colour-blind
     • For tips on colour selection for websites, visit
       • http://www.visibone.com/colorblind/

   

8. PROJECT CONCLUSIONS

   As a class we were pleased with the outcomes of the project. All of the teams were able to implement a working prototype for user testing. All of the students gained experience with a variety of user-centred design methods. Just as important, everyone (including the professor and teaching assistants) learned a great deal about the wealth of resources available through the UW Library web site. Our hope is that our design recommendations can be used to help make those resources more readily accessible to UW students, faculty, staff, and the community at large.

   Our thanks to the members of CNAG for their input into the individual team designs, and their overall enthusiasm for the project.

APPENDIX A Usability Heuristics

Norman’s Principles of Good Design For Understandability & Usability

[Adopted from: Norman, D. (1989) The Design of Everyday Things, New York: Doubleday]

Make Sure That....

1. Provide a GOOD CONCEPTUAL MODEL,
2. Make things VISIBLE,
3. Provide GOOD MAPPINGS
4. Provide APPROPRIATE FEEDBACK

Nielsen’s Ten Usability Heuristics

1. Visibility of system status. The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
2. Match between system and the real world. The system should speak the users’ language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
3. User control and freedom. Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
4. Consistency and standards. Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
5. Error prevention. Even better than good error messages is a careful design which prevents a problem from occurring in the first place.
6. Recognition rather than recall. Make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
7. Flexibility and efficiency of use. Accelerators—unseen by the novice user—may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
8. Aesthetic and minimalist design. Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
9. Help users recognize, diagnose, and recover from errors. Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
10. Help and documentation. Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large.

APPENDIX B – TEAM SUMMARIES

Team One

Team Two

Team Three

Team Four

Team Five

Team Six