Skip to the content of the web site.

Redesign of University of Waterloo’s Library Gateway

CLIENT CONTACT
Michelle Laing

PROJECT MANAGER
Scott Anderson

TEAM 5 MEMBERS

Ian Yi Dai 97246705

Phat Ha 98128232

General Leung 96015878

KJ Shipp 96126628

Melissa Thomas 98108461

 

Table of Contents

Executive Summary
  1. Introduction
    Interactive Systems Problem Statement
    Constraints 1.1.1 Visual Constraints
    1.1.2 Functional Constraints
    1.1.3 Performance Constriants    Requirements 1.1.4 Visual Requirements
    1.1.5 Functional Requirements
    1.1.6 Performance Requirements

  2. Phase 1 Design Methods
    Heuristic Evaluations
    2.1.1 Intent of Method
    2.1.2 Methodology
    2.1.3 Findings    Cognitive Walkthrough 2.1.4 Intent of Method
    2.1.5 Methodology
    2.1.6 Findings    Task Analysis 2.1.7 Intent of Method
    2.1.8 Method Specifics
    2.1.9 Findings    Competitive Analysis 2.1.10 Intent of Method
    2.1.11 Method Specifics
    2.1.12 Findings    Impact of phase one on design

  3. Phase 2 Design Methods
    Card Sorting Method
    3.1.1 Intent of Method
    3.1.2 Method Specifics
    3.1.3 Findings    Discount Usability 3.1.4 Intent of Method
    3.1.5 Method Specifics
    3.1.6 Findings    Impact of phase two on design

  4. Phase 3 Design Methods
    Keystroke Level Analysis
    4.1.1 Intent of Method
    4.1.2 Method Specifics
    4.1.3 Findings     Lab Based Usability Testing 4.1.4 Intent of Method
    4.1.5 Method Specifics
    4.1.6 Findings

  5. Recommendations

Appendix A - Team Project Task Allocations

Appendix B - Summary of Team Learning

Learning of Methods
Heuristic Analysis
Design Walkthrough
Card Sorting
Statistical Analysis
Learning in regards to Teams and Team Dynamics Appendix C - Summary of Client Minutes

Appendix D - Heuristic Analysis

Heuristics Used in Analysis
Interpretation of Heuristics
Conceptual Model
Mapping
Visibility
Feedback
Consistency and standards
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Help and Documentation
Findings

Appendix E – Cognitive Walkthrough

Results of Cognitive Walkthrough
User Testing Phase 1
Heuristic Evaluation 2:
User Testing Iteration 2:
Conclusions

Appendix F - Hierarchical Task Analysis

Use Library Forms
Use Find It
Use What’s New List
Summary

Appendix G - Competitive Analysis

University of Arizona
University of Washington
NCSU
Yale University
University of Pennsylvania
Dalhousie University

Appendix H – Phase 3 Results

Results for the Keystroke Level Analysis
Where would you go to find general help with a research topic in your department?
How do you get an article from a journal that is available at WLU?
Can alumni borrow books from UW?
What are the hours for the University Archives?
What books are available at the Kitchener Public Library?
Does the library have any electronic dictionaries?
Where can you find instruction for connecting from home?
Where would you go to renew books online?
What is the contact information for the Liaison Librarian assigned to your home department?

Appendix I – ORE Form

Table of Figures

Figure 1: Result for task completion time

Figure 2: Zoomed in on vertical axis

Figure 3: UW Library Gateway Prototype 1

Figure 4: UW Library Gateway Prototype 2

Figure 5: UW Library Gateway Prototype 3

 

Executive Summary

The university library acts as an important gateway in disseminating information to the university community. Therefore it is imperative that an easy to use gateway be designed for accessing library information. This system is to be used by graduate and undergraduate students, faculty members, library staff, alumni and community members.

To undertake this task, several phases of user testing have been completed, developing and modifying prototypes in a spiral design method. Through each iteration of design, several items are taken from the design phase and applied to the gateway to improve the usability of this site.

Many methods of evaluation were implemented during the first phase of design. Heuristic evaluations, where an expert user systematically inspects the user interface design for usability using a set of rules and axioms, was used as an initial evaluation technique. This evaluation was followed by a design walkthrough, where an expert user plays out some situations a user may encounter, and attempts to identify possible problem areas. A hierarchical task analysis was performed as well to determine what were the possible paths a user could take to solve a problem. Finally, competitive analysis was performed and ideas and techniques that worked well on other web sites were identified.

Through this phase of design, many usability issues were identified. One of the main issues that was identified was the burden on user memory that was required by the current design. This motivated the jump to a system where all the links were visible directly without having to interface through various links to see them. An additional mechanism that reinforced the decision to go to a text based system is that many of the various interfaces on the gateway were found to be inconsistent.

Through the heuristic evaluations it was found that the background graphic was distracting, and provided no benefit to the usability of the gateway. Through the competitive analysis it was thought that a direct Trellis search would be helpful and useful if placed directly on the gateway.

The next phase of design was undertaken at this point, and three different evaluation techniques were used. A card sorting method was used to attempt to model the user’s cognitive model of the situation. This method also developed a hierarchy of most important to least important links. Following this, discount usability testing was carried out, which involves a cycling of heuristic evaluations followed by design walkthroughs.

These analyses revealed that the electronic resources was the primary destination of most library users who participated in the user testing. As an additional comment most users complained about the lack of graphics in the design of the gateway,

As the final phase of design, a keystroke level analysis was done for both a novice and expert user. This method attempts to quantify how long a novice or advanced user would take to complete a given task. These numbers were used as benchmarks in a phase of user testing that followed.

From this phase, several errors in consistency were found, and the users still wanted to see more graphics on the gateway. Statistically, there was no significant improvement on access time of the gateway just designed and the previous UW library gateway, except for in one scenario. However, the mean time to complete the task using the current gateway, fell within the error tolerance of the means measured in the study.

It is recommended that the mouse over technique be eliminated and changed to a text based menu system. This will improve the use by novice users as they will have all their options presented to them. The hierarchy and diction of the menu systems should be altered to better fit with the user’s mental model of the problem. The background graphic should be removed as it was found to be distracting from the text and also increased the loading time. As well, a quick search Trellis bar should be added to the gateway page. Through the user testing it was noted that students primarily used the library gateway page to find the location and call number of information in the library.

  1. Introduction

    2. Advancements in information compilation and dissemination systems have made it possible to manage an ever growing base of knowledge. Without access to this knowledge research, institutions such as the University of Waterloo (UW) would begin to stagnate as scientific advancements are allowed to pass without an opportunity for their scrutiny.

    The university library acts as an important gateway in disseminating this information to the university community. Currently implemented, the university has a web based gateway that allows for access to the current collection of both printed resources, as well as other services provided by the library. However, this gateway has not been designed with the end user in mind, and thus lacks the intuitive usability that would make it a truly useful tool across the entire anticipated user base. It is thus necessary to improve the web based gateway to allow ease of access for all students, faculty, and other members of the university community for the purpose of research.

    In this report, various methods of usability evaluation will be covered as they apply to the University of Waterloo Library’s gateway page. From the result of these evaluations, recommendations on possible changes to the current gateway page will be made. Finally, a prototype of a functional web page will be developed and rationalized.

    Interactive Systems Problem Statement

    Design a library gateway for easy use, for accessing library information to be used by graduate and undergraduate students, faculty members, library staff, alumni and community members.

    To develop this statement, the group developed a matrix that listed various facts, assumptions regarding this problem statement. This is included below in

      User Activity Users Level of Support Form of Solution
    Facts Access trellis All users Help available for new users, know what they are looking for Trellis link, quick search option, help link
    Search for information on variety of subjects Students, community members Low, intuitive, know general subject area Information by subject link, e-journal/data/text, journal index links
    Access gateway from home Students and faculty Familiar with internet Help section available for this function
    Access and borrow information from UG or WLU Students and faculty Low Trellis lists material at those libraries, direct links to borrowing info from them
    Assumptions Electronic information accessed most frequently Students, faculty Low, automatic All electronic links shown on gateway
    Users won’t take the time to read What’s New Students, faculty Low, very visible Information present on the main gateway
    Constraints Must be accessible on all browser types, All users Low Primarily text
    Requirements Must have the library, trellis link, e-journals/data/text, All users Low Ensure that they are on gateway

    Table 1: Interactive Systems Problem Statement Matrix

    This gateway page will be the first thing that users see when visiting the UW library. It will link users to all the available information concerning the UW library as well as information on other libraries.

    One of the primary criteria that the gateway must provide is a link to the Trellis search site. All varieties of users walking into the library to use their services will access the web page and need the services to Trellis, and it is reasonable that this is a requirement for a solution to be considered feasible.

    The gateway page should load quickly and all items on the page should work with all Internet browser programs since users may access this gateway from outside the school, and these computers may not be equipped with the same browsers as the library.

    The library does offer courses that introduce new users to the gateway page as well as other aspects of the library but the majority of students do not make use of this service. Therefore it is important that the gateway be intuitive and that users can walk up and use the page.

    A variety of users with different experience will be using this page so it must be designed to meet the needs of the majority. Librarians and faculty members will likely be more expert users and would like to have quick and easy access to links that they commonly use. Students and community members are likely to be novice users and therefore it is important to have clearly labelled links.

    The gateway can be used to access a great amount of information. A common use for the page is to access the library catalogue and perform journal searches. Students will commonly use the library gateway page to search for items on research projects. For novice users it will be helpful to have links to research information that has been sorted according to subject. This link will provide them with a starting point to familiarise them with different options that are available to be used in research. Links to searching journals and texts based on subject topics are important to have. UW library is connected to the Guelph library and Wilfred Laurier library so that students may borrow resources from all three of the libraries. Other local libraries are also useful to have access to as an alternate source of gathering research information.

    A quick link to the UW catalogue Trellis is essential on the gateway page and it should be easily accessible, as most users of the library site is interested in finding information on Trellis catalogue.

    The library gateway should offer help for the users on what information is on the page, how to find things on the site, help on research and how to connect form home. Access to information on how to borrow from the other libraries needs to be present and access to the forms required to request information from the other libraries should be provided. The library gateway should provide adequate help information for a variety of users. Information such as how to connect to the library server from home should be easily obtained. Other help information for the library and its resources should be available as well through the gateway page.

    Hours of the library operation hours are important for users to have access to as well as the location of the libraries for new users and users from the community. Services that the libraries offer are important to have for all users to allow them to discover what services are available to them through the library. Distance education students are generally not on campus so it is important for them to have access to the library from home and all the research tools that are available on campus.

    Users often borrow information from the library and then may lose the information slip with the due date on it, the library page should provide a method of access to patron information. The patron information should contain information on due dates of material, overdue materials and see the status of any material you placed a request on. Another feature that would be useful to have especially when books are overdue is an option to renew material online.

    Often time professors place course material on reserve in the library for students to access. Access to the location of the location and call number of this information is important to have on the gateway page. Other course information such as course texts will also be helpful for the students.

    A "What’s New" section is important to have on the gateway page to make users aware of new services or changes that the library offers. The sub menu items should be displayed on the main page as users will likely not take the time to search for these or connect to the "What’s New" link. If the sub menu items are of interest to the users then they may connect to the link and find out more detailed information.

    Information on the services offered by the library should be displayed somewhere in the library site. A link to the services is necessary and information pertaining services for students, alumni, faculty members should be accessible through the web site.

    The above descriptions on the system needs illustrate a general outline as what the system needs to support the activities of the targeted users.

    Constraints

    The constraints on the UW Library gateway are the boundaries and limits which must be met for the gateway to be considers a feasible solution. Visual constraints are the constraints on the visual aspects of the screen including colour, fonts size, etc. The Functional constraints are limits on the functions the page can include and Performance constraints are the boundaries of the technical aspects of the gateway.

    1.1.1 Visual Constraints

    • No more then 4 different colours of text.
    • No more then 256 colour graphics.
    • The font size must be set no less then SIZE="-1"

    1.1.2 Functional Constraints

    • The interface must not provide any audio feedback.

    1.1.3 Performance Constriants

    -The page must fit on a screen with a resolution of 800X600 pixels.

    Requirements

    Requirements are what the design must accomplish to be considered a successful design. Visual requirements are those that detail the necessary graphical details as well as the "look and feel" of the interface. Functional requirements detail the minimum functionality the interface must be able to provide in order to be considered a feasible solution. Performance requirements are the technical aspects of the gateway.

    1.1.4 Visual Requirements

    • Must include the UW Logo
    • The word Library must appear on the page
    • The text must be high contrast

    1.1.5 Functional Requirements

    The user must be able to access:
    • TRELLIS from the main page
    • Course Reserves
    • The UW main home page
    • The Wilfred Laurier library page
    • The Guelph University library page
    • The TUG home page
    • The Electronic Journal Search engine
    • The electronic Journal indexes
    • Electronic Text index
    • Electronic Data index
    • Reference Tools
    • Patron Information
    • Library hours and recent news
    • Access to help and other instructions
    The user must be able to access from the main page the following forms:
    • TUGdoc services
    • Inter-Library Loan
    • CISTISource
    • Reserves
    Feedback to the user must be provided
    • A text box must appear when the user move the mouse over a graphical hyperlink
    • The link address must be shown in the web browsers status bar

    1.1.6 Performance Requirements

    • The page must dynamically adjust when the window size changes
    • On a library terminal the page must load in less then 2 seconds
    • For off-campus users, using a 56K modem connection the page must load in less then 15 seconds
    • The page must function using both Microsoft Internet Explorer and Netscape Navigator
    • The page must run with a minimum amount of maintenance

  2. Phase 1 Design Methods

    3. Heuristic Evaluations

    2.1.1 Intent of Method

    A heuristic evaluation is an informal process whereby an expert user judges the interface using a set of usability axioms, or ‘heuristics’. The expert user systematically inspects the user interface design for usability using these rules and through this, finds usability problems with the design. Once these problems are found, they can be addressed in the iterative design process. It is by far one of the most popular usability methods because of its apparent ease of use and intuitiveness.

    2.1.2 Methodology

    For this study, Norman’s four principles and Nielson’s ten usability heuristics were used to perform the evaluation of the walkthrough on the UW library gateway. These heuristics and principles are listed in Appendix D - Heuristic Analysis.

    The research group divided into two sections and one section collaboratively analysed the UW library gateway page using Norman’s principles. The other section collaborated and evaluated the same page using Neilson’s ten heuristics. In splitting into two sections and collaborating within these sections, the research group attempted to capture more usability issues by having more individuals trading ideas. Thus once one individual notices one usability problem, the remainder of the group can add to, and build on these ideas.

    Additionally, it was decided before doing the evaluation, that several of the heuristics were not applicable to the design of a web based interface. These heuristics were eliminated from the evaluation. Since the heuristics leave a lot up to the interpretation of the expert users, the interpretation of these heuristics was agreed upon within the design section before the evaluation of the web page.

    A brief outline of the research group’s interpretation of each of the heuristics can be found in Appendix D - Heuristic Analysis.

    2.1.3 Findings

    A summary of all the findings for each heuristic is included in Appendix D - Heuristic Analysis.

    On the whole, the gateway satisfies many of the heuristic criteria to some degree. However, where the gateway is most lacking is in the consistency of the interface throughout the page, as well as the amount of memorization that needs to be done. The various interfaces, from the mouse over on the main access buttons, to the drop down and go in the advanced user menu, to simply clicking on the button bar, needs to be standardised to present a more intuitive user interface.

    Additionally, there must be some better way to display the menus such that the user need not memorize the links that are embedded within each pop window.

    Cognitive Walkthrough

    2.1.4 Intent of Method

    A cognitive walkthrough is a user centred design method that is completed by a usability expert without the involvement of users. This type of walkthrough is done by designing several common tasks that must be completed by users using a certain product. The expert evaluator then performs these tasks and during this process can identify where the users may run into difficulties or problems. This simulation of the user problem-solving process will single out areas that need to be improved, as well as identify areas that the expert evaluator thinks should be within the knowledge of the user. Using simulation of real problems, as opposed to simply delving into the various problems the user may encounter, structures the inspection as well as reveals some situations that users will actually run into in the use of the product.

    2.1.5 Methodology

    For the purpose of evaluating the gateway, a set of twenty questions were used for the walk through process. These questions are included in this report in Appendix E – Cognitive Walkthrough. Each individual within the design team read through the questions and then found the most direct and obvious method to get to the information that was required. After this route was found, the evaluator back tracked and stepped slowly through the process, noting any significant areas where problems may have been encountered. Any problems encountered during the completion of the task were recorded and comments on what areas could use improvement or what areas were working well in the current design. In particular, the evaluators attempted to find difficulties in wording, confusion in the navigation, as well as identify if objects and interfaces one the gateway were visible. Each individual evaluator needed to bring their own interpretation of these criteria when evaluating the gateway.

    2.1.6 Findings

    One of the positive findings of the cognitive walkthroughs is that many redundant pathways are available to complete several tasks. This can actually assist a user in accomplishing their end goals, as the same link is available in several different locations. All of these locations are logical, and the design team can identify how they fit in under various users’ cognitive models.

    One of the problems encountered was the Library Forms menu. It is a drop down type and only one of the menu items is initially visible to the user. Selecting an item does not automatically link the user to the selected menu item, rather, a redundant button press is needed.

    Another area that requires redesign is identified when attempting to locate the link to the UW Bookstore. Since this enterprise was considered to be a part of the university community, an attempt to locate it was first made under Local Sites. Searching was also used, but proved to be unsuccessful, in trying to obtain information on accessibility issues for the visually impaired.

    The inconsistent layout and overall presentation of each underlying page forces the user to adopt a different conceptual model of how information is laid out on that page. This increases the time needed to scan for the correct information and forces the user to correctly interpret the link descriptions that do not seem to be consistently applied.

    Task Analysis

    2.1.7 Intent of Method

    Task analysis is an approach for modelling the interaction between a device and a user and has been an established technique for over 25 years. Using this method, the research team can identify pathways to solve certain problems, and begin to identify what may go wrong during the solution of the problem.

    2.1.8 Method Specifics

    To perform this portion of the evaluation, several tasks were selected from the main page that the design team thought may have been used by users. At each level, the various steps that could be taken were evaluated and noted. If there were several different paths that could be taken from a method, they were all listed, however only the first path, as chosen by the design team, was followed through. A list of the different steps, down to the detail level of a click, were recorded and noted in the task analysis flow.

    2.1.9 Findings

    The entire task analysis portion can be found in Appendix F - Hierarchical Task Analysis. There are three ways to get the ‘What’s new’ floating list to appear. The problem is that the location of the list is not clearly connected to any of the ways of making the list appear. The button at the top looks like a link and should act like a link, but instead it just reloads the page. Resetting the list by moving the pointer over the bottom of the page was discovered by random searching. There should be a more intuitive way of retrieving the ‘What’s new’ list, or the list should be available at all times.

    From the task analysis it can be seen that getting to any of the available library pages can be done with a minimum of steps, but the methods of getting the lists of links is inconsistent and in some cases confusing. A more consistent and clear method should be used to link the main title and the sub-titles associated with it

    Competitive Analysis

    2.1.10 Intent of Method

    The intent of competitive analysis is to examine similar library gateways of different universities across North America and extract superior elements of their designs. Also the current design of UW library gateway is compared with others to study the pros and cons of the current systems.

    2.1.11 Method Specifics

    A group of six universities were chosen from among the list provided by the clients.
    • University of Arizona
    • University of Washington
    • NCSU
    • Yale University
    • University of Pennsylvania
    • Dalhousie University

    For each of the universities, one of the design team members was responsible to evaluate the library gateway page using the Norman's principles and Nielson's heuristics. The same definition and criteria as defined in the previous section was carried forward to this evaluation. A comparative analysis with the UW design was performed where direct comparison over each heuristic was made and conclusions on the positives and the negatives of the current gateway were proposed.

    2.1.12 Findings

    The conceptual model of UW site differs substantially from the majority of the other library site we evaluated. UW site uses a "click and show" model that is used in one other university library page we studied. Most sites would display all the links and textual information on the gateway page, with nothing hidden. The mapping criteria of current UW site are met well. Some other universities do not have a clear and organized gateway mapping. However, many other sites evaluated contain a quick search function on the gateway page, whereas UW site does not. In terms of feedbacks, UW gateway page has done well in general compared to the others. It provides adequate feedback to the users with regard to its "mouse over" link access mechanism. There are visibility issues on UW sites in comparison with other sites. The UW gateway page is quite inconsistent with the rest of the library pages and university pages. The loading time of the background graphics is worth investigating as well.

    Please see the detailed findings/results can be found in Appendix G - Competitive Analysis.

    Impact of phase one on design

    Through this phase of design, many usability issues came up that were identified via the various usability analyses performed on the gateway. As evidenced through our findings, many of the usability techniques produce similar overlapping results. It is upon these areas that the designers need to focus. Several of the key findings are re-iterated below.

    One of the main issues that was identified was the burden on user memory that was required by the current design. Too much of the information was hidden behind the scroll over menu’s. Since it was found that it is not very much faster to go through the advanced user menu’s, the information contained within the pop up menus needs to be remembered by even the expert user in order to efficiently use the system. This motivated the jump to a system where all the links were visible directly without having to interface through various links to see them. Not only does this provide quicker access time to all the links, it also shows a novice user their entire range of options.

    An additional mechanism that reinforced the decision to go to a text based system is that many of the various interfaces on the gateway were found to be inconsistent. The drop down menu, the menu bar, and the scroll over buttons all required a different method of interfacing. Thus by going to a completely link-based system, these interfaces are kept consistent across the entire gateway, further reducing the mental load required of the user.

    Through the heuristic analysis it was found that the background graphic was distracting, but provided no benefit to the usability of the gateway, and in fact hinders usability since the link to the text version is nearly completely obscured by the background. Since the purpose of the redesign is to improve the usability of this gateway, the graphic serves no purpose to further this goal, and thus was removed.

    Through personal experience of the research team, it was thought that a direct Trellis search option would be helpful and useful if placed directly on the gateway. It was rationalised that most users enter the library to find holdings within the library, and these are two things that should be intimately related. Thus, when a user opens the library web page, they would expect to find searching capabilities. These are not there in the current design, and were proposed for the next prototype.

  3. Phase 2 Design Methods

    4. Card Sorting Method

    3.1.1 Intent of Method

    Card sorting was used to try to identify clearer headings for the groups and to identify which menu items should be grouped together. This technique is best used in the early stages of development to determine how users would group various functions. Card sorting is used to give the designer an idea of the user’s mental model. Concepts which users place in the same category show an association between those categories in the user’s mental model of the system. These should then be logically grouped in these methods to encourage compatibility between the system and the model.

    3.1.2 Method Specifics

    It was felt that some of the menu names on the original UW gateway were not very intuitive. Thus it was desired that the users come up with some titles that would relate information contained within that group. A free form method of card sorting was used, in which the users decide how they are going to group the links together and how many groups to use.

    To perform the card sorting method all of the current menu items on the UW gateway page were taken and written out on individual pieces of paper.

    These are listed below in no particular order, for completeness:

    TRELLIS

    Services for Persons with Disabilities

    Other Library Catalogues

    Distance Education

    E-Journals

    Alumni

    E-texts

    Business Community

    E-Database

    UW Home Page

    By Subject

    Laurier Library

    Internet Search Tools

    Guelph Library

    Reference Tools

    Tug Home Page

    Get from Waterloo

    TUGdoc

    Get from Guelph/Laurier/Annex

    ILL

    Get from other Libraries

    CISTISource

    Renewals

    Self Registration

    Course Reserves

    Reserves

    View your record

    Ask Us/Tell Us

    News/Events/Exhibits

    Buy It

    Hours/Locations

    TRELLIS Help

    Guide to the Libraries

    Research Guides

    Staff Administration

    Tours and Workshops

    Library Development

    Online Instruction

    Accessibility

    Site Index and Search

    Services for Faculty/Staff

    Connect from Home

    Services for Graduate Students

    Services for Undergraduates


    Users were selected at random amongst those that were using the library system on the day of the testing. Users were asked if they would like to participate in a brief study taking five minutes of their time. The user was given the option not to accept, however if they accepted, the user was taken to a cubicle in the library. Then the above cards were given to the user along with some blank pages. The user was told that these items were all for the UW library and were then instructed to sort the cards into logical groups and name each of the groups. If they felt anything was missing they wrote this in on the blank cards and put it in the appropriate grouping. They also prioritised the groupings from most used to least used. Consistency between the groupings and prioritisation were analysed. From this the links were grouped together and a title was generated that incorporated the information and words conveyed in the titles developed by the users.

    3.1.3 Findings

    The users grouped the links into the following headings, and under each heading a link in descending order of importance (with the most important links appearing at the top of the list).

    About the Library News/Events/Exhibits
    Hours/Locations
    Guide to the Libraries
    Staff Administration
    Library Development

    Services For Faculty/Staff
    Graduate Students
    Undergraduates
    Person's with Disabilities
    Person's taking Distance Education
    Alumni
    Business Community

    Borrowing MaterialsFrom Waterloo
    From Guelph/Laurier/Annex
    From other Libraries

    Course Information

    Reserves
    Book Look

    Renewals

    View Patron Information

    Electronic Resources

    Library Catalogues
    Journal Indexes
    E-Journals
    E-text
    E-Data
    Catalogues
    Research Guides by Subject
    Internet Search Tools
    Reference Tools

    Library Forms

    TUG Doc
    ILL
    CISTISource
    Self Registration

    This information should be used and the groupings used as above as this is the model which is most intuitive to a the number of users tested.

    Discount Usability

    3.1.4 Intent of Method

    Discount usability is a method which requires few resources but gives the benefits of usability testing. It combines the use of heuristic evaluations and design walkthroughs. According to Neilson very few users are needed in discount usability, usually by the time five users have been tested you have identified about 80% of your problems. Discount usability has a better ratio of benefits to costs then more formal methods of usability testing. Discount usability combines the use of heuristic evaluations and design walkthroughs. Heuristic evaluations are good as the designers can perform them quickly. A drawback is that it is a designer who is performing the test and they will not see the system in the same perspective as the users will. Design walkthroughs have the benefit of having actual users perform the testing. This method will also give some insight into what the users mental models are when they say why they did things certain ways. Discount usability allowed us to maximize our usability testing with a minimum number of users being tested.

    3.1.5 Method Specifics

    In the previous iteration of design, nine heuristics were selected. The same nine heuristics were used from Norman and Neilson to evaluate the prototypes of the UW site. These same heuristics were previously used to evaluate the other university library web sites. The heuristics used were conceptual model, mapping, visibility, feedback, consistency and standards, recognition rather than recall, flexibility and efficiency of use, aesthetic and minimalist design and help and documentation.

    From the card sorting (see above) a second version of the prototype was developed. A heuristic evaluation was then performed on this prototype. The research group split into two pairs and performed a heuristic evaluation of the second version of prototype using the nine specified heuristics. The prototype was then modified to reflect the problems encountered during the heuristic evaluation. Two users were then tested using design walkthroughs. Ten questions were selected from the twenty used in the initial evaluation of the current UW gateway page that was thought to best represent the widest range of functions. The ten questions were:

    Design Walkthrough Questions

    1. Is the University Map and Design Library open on weekends?
    2. I need to contact the librarian for my department – how do I find his/her phone number or e-mail?
    3. Does the Library have any electronic dictionaries?
    4. Where can I find a database in which to locate articles on Anthropology subjects?
    5. Guelph Library has the book I need. How do I have it brought to UW?
    6. How can I discover if any materials have been placed on Reserve for my courses?
    7. How can I discover when my UW library books are due?
    8. How can I find out if my course textbook is in stock at the UW Bookstore?
    9. What library orientation sessions are being offered for graduate students this term?
    10. Where can I find instructions about connecting from home?

    A low-fidelity paper prototype was used. One member of the research group read the questions to the user and the user pointed to or said which link they would use to answer the question. If the link selected would take them to a menu that was on the original UW gateway web page another sheet of paper with the corresponding menu list was shown to the user and they selected the item from there to answer the question. The second member of the research group recorded what steps the users took and how long it took them to select the link. The same member was used to evaluate all users and also rated the time to select the link as quick, medium or slow. The user was evaluated to have responded quickly if they were able to identify and name the link within five seconds of asking the question. The medium response time was given for those users who found the link between six and fifteen seconds. Slow was all the responses after fifteen seconds. These were arbitrarily set values, and are only as a means of comparison between the different response times given by different users.

    The prototype was then modified again and a heuristic evaluation was completed by two groups of two again using the same nine heuristics. From these findings the prototype was modified again. Two users performed a design walkthrough to evaluate the new version of the prototype. The same 10 questions were used as were used in the first trial. From this information the final draft prototype was developed.

    3.1.6 Findings

    A complete description of findings for this method can be found in Appendix E – Cognitive Walkthrough, Discount Usability Findings.

    First Prototype:

    From the first user testing iteration, it was noted that the users have consistently shown trouble with identifying the services for graduate students and finding a link to an electronic dictionary. The majority of users point to News and Events under About the Library to look for the services and have given up looking for the dictionary link.

    Most users believe the small text underneath is a quick link directly to the web page, whereas it points to the same link as the headings. The users have commented on the general easy accessibility of the proposed gateway page. Some expressed concerns over the plain nature of the heading, graphics are suggested for the next prototype.

    Second Prototype:

    In the second and final prototype of the discount usability method iteration, it was observed that the time to complete the task and the accuracy in answering the questions were improved over previous prototypes. However, from the general comments received during testing, this prototype is still lacking in graphical design. To increase the visibility of the headings and the visual appeal of the web page, icons were intended to replace major headings in the next iteration of design.

    As for other minor changes, the sub menu items under the headings which were currently just one link should be broken into separate links that directly link to underlying pages. This should increase the efficiency of the gateway page and allow faster access for experienced users. The Trellis link should incorporate an icon and text, and should be displayed on top of the gateway for instant access to the UW library catalogue.

    The prototypes for Phase 2 can be found in Appendix E – Cognitive Walkthrough, under Prototypes in discount usability testing.

    Impact of phase two on design

    As noted above, many areas of the gateway still needed to be improved. The most salient errors and areas for improvement are summarized.

    In the second iteration of design, one of the major accomplishments was developing a hierarchy of menu items based on the results of the card sorting method performed. In addition, another finding from this evaluation method was that the menu titles were not intuitive.

    Several iterations of user testing and feedback revealed that the layout of the page was too crowded, and attempts have been made to remove some of the links from the main page in order to reduce the clutter. Furthermore, these discussions also revealed that the electronic resources was the primary destination of most library users who participated in the user testing. In addition, most users complained about the lack of graphics in the design of the gateway,

  4. Phase 3 Design Methods

    5. Keystroke Level Analysis

    4.1.1 Intent of Method

    Keystroke level analysis method is used to predict the time required to perform a task without any errors. The task is broken down into component activities (Stanton, 1998). There are four mental operators, keystroking, pointing, homing, drawing, one mental operator and one system response operator. The keystroking time varies depending on the typing ability of the user. The time required for the five previously mentioned components is calculated and summed to give the total predicted time required to complete the task. The time for each of the operators has been experimentally derived (Stanton, 1998). This method was designed for expert users so it does not give a very good estimate for the time novice users will require. Also no errors are calculated into the task time. Some error time could be calculated by predicting the errors the user may make and calculating the time required for them to fix their error. The mental preparation time is very difficult to incorporate into the equation, as there may discrepancies in where the mental preparations times should be in the task. These predicted times are used to set benchmarks for the users.

    The purpose of performing lab based usability was to test a variety of users and see what kind of problems they encountered when using the new library gateway page. This was also a way to see if the benchmarks set by the KLA could be met by the users. A variety of users of different levels of experience and from different faculties will provide new insight into the web page that the design team may not have anticipated. One limitation in the variety of users was that the participants were friends of the designers and were likely from the same faculties and year of studies. Thus they would have similar experience with the library web page. It was noted one of the librarians that often the arts and environmental studies students encounter different problems then the engineering students. This may not have been accurately tested as these faculties were likely not equally represented in the testing sample.

    4.1.2 Method Specifics

    Keystroke level analysis (KLA) was performed by one member of the group to maintain consistency. The different possible routes to access the information to the 10 questions provided were looked at. For the most direct route the KLA was also done for a more novice user who is not familiar with the layout of the web page and would need to visually scan the page to find the correct link. The items in the top bar of the page were assumed to be in groups of six chunks of text that the user would scan. It was assumed that the users would scan from left to right and top to bottom. They would initially scan the bar across the top of the page and then go through each of the three columns of the web page.

    4.1.3 Findings

    The findings for this analysis can be found in their entirety in Appendix H – Phase 3 Results. The times were tabulated for both the expert user (T1) and the novice user (T2) and are summarised below in Table 2.

    Question

    T1

    T2

    Where would you go to find general help with a research topic in your department.

    2.65

    15.65

    How do you get an article from a journal that is available at WLU

    2.65

    14.65

    Can alumni borrow books from UW?

    2.65

    11.65

    What are the hours for the University Archive

    2.65

    13.65

    What books are available at the KPL?

    2.65

    18.65

    Does the library have any electronic dictionaries?

    2.65

    17.65

    Does the UW Library have a copy of Andrew Pyper’s Lost Girls?

    2.65

    5.65

    Where can you find instruction for connecting from home?

    2.35

    16.35

    Where would you go to renew books online?

    2.65

    14.65

    What is the contact information for the Liaison Librarian assigned to your home department?

    1.65

    14.65


    Table 2: Summary of Results for KLA

    These results are used in the following section of design to compare actual response time of users with the theoretical response times calculated here.

    Lab Based Usability Testing

    4.1.4 Intent of Method

    Laboratory usability testing will provide a quantitative measure of the design modifications implemented thus far. External validity is strengthened through the use of a high fidelity prototype, assessing tasks that are representative of common library uses, selecting participants from the university community, and locating the test in an environment familiar to users, i.e. an on campus computer lab. A comparison of user performance with respect to benchmarks, generated using KLA, for novice users, expert users, and those derived from user testing on the current UW gateway will facilitate the identification of design deficiencies, and thus, possible directions for future redesign.

    Laboratory based testing of gateway was conducted in the past by Nielson. In his study at Sun labs Nielson allowed users to explore the current iteration of a design and then assessed their performance in the completion of a task. Unlike the present study, Nielson employed a one way mirror and several cameras to assess the performance of users unobtrusively. One redesign resulting from user testing was followed by an increase in site usage by 416 percent compared to a server activity increase of 48 percent.

    4.1.5 Method Specifics

    The office of research ethics form completed and approved is included in Appendix I – ORE Form

    Nine participants, 5 male and 4 female, were selected on a voluntary basis. Participant ages ranged from 20 to 37 years and were enrolled in undergraduate and graduate programs in the Faculties of Engineering and Applied Health Sciences at the University of Waterloo. All participants indicated that they were familiar with everyday computing activities, such as, word processing and email, and the use of web pages for online applications. Two thirds of users felt familiar with UW Library services and four users were familiar with the design and creation of web pages. Seven users were familiar with UW Library web pages.

    Prior to participation in the user trials, users completed a survey concerning their familiarity of the UW library gateway and general computer uses.

    Users were seated in front of a computer displaying the prototype on Internet Explorer 5.5 (IE). Participants started with their hand on the mouse and the mouse cursor placed at the bottom centre of the visible screen. Two sequences, Version A and Version B, of tasks were used and selected on an alternating basis. Participants were read the task question twice and were instructed to attempt to complete the task as quickly as possible after the question was read for the second time. Task completion time was measured from the point that the question was finished being read for the second time to the point where the user clicked the mouse button on the correct task completion link. These task completion links were determined before the user testing and can be perused in Appendix H – Phase 3 Results. After each task, the participants returned the mouse cursor to the bottom centre of the screen and kept their hand on the mouse. All indications of previously accessed links were erased with the use of the history delete history feature of IE. After all tasks were completed participants completed a usability questionnaire based on their experience with the prototype (see Appendix I – ORE Form). Participants were encouraged to complete all tasks to the extent that their mental well being was not compromised. In such cases the participant’s attempt was halted and their results were valued as null.

    Background questions were compressed by aggregating responses from 3 to 5 as ‘familiar’ and 1 to 2 as ‘unfamiliar’. Response 3 was placed in the ‘familiar’ category because the question explicitly uses the word ‘familiar’ which may bias the user towards familiar. Similarity, responses from the usability questionnaire were compressed into user-friendly and not user-friendly by compressing 1 to 2 into the former category and 3 to 5 into the latter. As with the compression of the familiarity question, compression of the usability questions was based on the explicit use of the word ‘user-friendly’ in the question.

    4.1.6 Findings

    A complete list of the statistics can be found in Appendix H – Phase 3 Results. Task Completion Times

    Table 3 Significance Matrix of User Performance and Benchmarks

    Task

    Benchmark

    KLA Expert

    KLA Novice

    KLA Current

    1

    Y

    Y

    N

    2

    Y

    N

    N

    3

    Y

    Y

    N

    4

    Y

    N

    N

    5

    Y

    N

    N

    6

    Y

    N

    N

    7

    N

    N

    N

    8

    N

    N

    N

    9

    Y

    Y

    Y

    10

    Y

    N

    N


     

    Mean user performance on tasks 2, 4, 8, and 9 was better than all three benchmarks – only performance on task 9 was significantly different with respect to benchmarks set for expert and novice users and those derived from the current UW gateway (t(df = 8, p = 0.05) = 1.88, t(df = 8, p = 0.05) = -4.44, t(df = 8, p = 0.05) = -5.9, respectively). All other measures were not statistically significant at the p = 0.05 level.

    Although not statistically significant, eight respondents rated the prototype as user-friendly with respect to keyword, major links, and navigation bars. Six felt that the graphics improved the prototype’s usability and seven reported that the overall layout was user-friendly. Seven out of nine respondents rated the prototype as better than the current UW gateway. Table 4 presents the responses from the usability questionnaire.

    Table 4 - Summary of Usability Questionnaire Responses

    		Not user-friendly User-friendly

    chi

    chi critical

    significant

    Key words

    1

    8

    3.130

    3.842

    No

    Major links

    1

    8

    3.130

    3.842

    No

    Graphics

    3

    6

    0.167

    3.842

    No

    Navigation bars

    1

    8

    3.130

    3.842

    No

    Overall layout

    2

    7

    1.185

    3.842

    No

     

    Better than UW

    Not better than UW

    Overall comparison

    7

    2

    3.842

    No

    p = 0.05

    Task 3 and task 7 contained extreme scores which where much greater than the score directly below it. Although these scores limited the conclusions drawn with respect to these tasks they were not eliminated because proper testing protocol was used in their collection. Mean performance on tasks 1, 5, 6, 7, and 9 were greater than the novice benchmark, hover, the lower 95th confidence interval encompassed the novice score.

  5. Recommendations

    6. The mouse over menu technique should be eliminated and changed to a text based menu system. The mouse over menus were found to place a large burden on the users memory as the information is not always present on the screen. This will improve the use by novice users as they will be required to remember each of the items that were under one menu title when searching for something. Experts also will not have to remember the location of the menu items.

    Through card sorting the groupings of the menu items was altered from the original gateway page. The "services for" link was moved to become a sub menu of "About the Library". All of the information previously contained in the "services for" link are now accessed through an underlying off of the main gateway page. Course reserves previously were contained under "get it" but the majority of users felt that it should be a separate category. View Patron was also not associated with "get it" and thus a new category was created for information concerning patron information.

    Through heuristic evaluation and cognitive walkthroughs it was identified that some of the menu titles were not very intuitive. Through card sorting new more intuitive menu names were developed based on user feedback. Find It should be changed to Electronic Resources and Get It should be changed to Borrowing Materials. These new titles were developed directly from the users.

    Originally the link for local libraries had been eliminated from prototypes one and two, however the client felt that this was an important link so it was added back onto the gateway. This link was relocated from local sites to electronic resources.

    The background graphic was removed as it was found to be distracting from the text and also increased the loading time. The loading time is critical as the library page can be accessed from home and users often become impatient while waiting (www.useit.com). Users at home will have a large variation in speeds of their internet access and also have a variety of internet browser programs which may affect the clarity of the graphic. The graphic also hides the text version link on the current UW gateway page.

    A quick search Trellis bar should be added to the gateway page. Through the user testing it was noted that students primarily used the library gateway page to find the location and call number of information in the library. It was felt that a direct link to do this on the main page would be beneficial to students and other users. If new users are not familiar with what Trellis is they will be more likely to recognize the search function. The UW site search was added on as well to allow users the ability to search a larger area from the library page.

    The page was divided into three main columns to try and reduce some of the clutter. The quick search tools and the logo’s were visually separated from the rest of the page with a horizontal line. The menu bar on the bottom was also separated from the body through the use of a horizontal bar. Electronic Resources were found to be the most commonly accessed menu items based on user comments thus each of the menu items were included as separate links under the title. This section was also placed in the left most column of the page as users generally read from left to right and top to bottom. The middle column contains other important links that were not noted to be used as frequently as the electronic resources. Library Forms was placed in the right most column as it is generally used by expert users who know exactly where to look for it. Novice are not likely to understand the abbreviations used in library forms but the expert users like to have a direct link to the information that they require. The links available for advanced users has been altered from a drop down menu into a list. This menu still allows advanced users to quickly access their desired sub menu, if not improving the access as they do not have to click on a ‘go’ button, yet thematically fits with the rest of the page.

    It was felt that the What’s New category was important for the library to convey information on any changes taking place. All of the submenu items are displayed as users are not likely to take the time to go into the link and search for what’s new at the library. This way the new items are presented on the screen and if the users are interested in the item they can go directly to the link and obtain more information.

    The menu bar at the bottom of the screen was added to give users a help function and a link to the library gateway and the UW homepage. It was designed to take the place of the menu bar that is currently on the top on the UW gateway. The current menu bar has three link buttons on it but it is not clear that they are links. This new menu bar will ideally be displayed on all of the library web pages. On the underlying pages it will be most useful in providing help and a method to undo errors by returning to the gateway. It placed on the gateway page to provide consistency between all of the pages.

    A graphic was added to the top bar of the page to add some aesthetic appeal to the page without creating a large distraction from the information. The idea of using icons to accentuate the menu titles was contemplated but it was difficult to develop icons that would intuitively demonstrate what they would link to.

 

Appendix A - Team Project Task Allocations

Ian Dai

Report 1:
Heuristic evaluation of current UW library gateway using Norman’s heuristics, cognitive walkthrough of questions 13 to 16 that were provided by the CNAG group, competitive analysis of Yale library website, draft specifications, proposal for changes to current site, develop prototype 1

Report 2:
Discount usability testing, develop updated prototypes, findings from discount usability

Report 3:
Editing and modifying report 2 methods, lab based usability testing

Phat Ha

Report 1:
Heuristic evaluation of current UW library gateway using Neilson’s 10 heuristics, cognitive walkthrough of questions 1 to 4 that were provided by the CNAG group, competitive analysis of U of Washington library website

Report 2:
Description of low fidelity prototype, discount usability testing, develop prototypes, findings of discount usability

Report 3:
Statistical analysis, results from statistics, lab based usability testing

General Leung

Report 1:
Heuristic evaluation of current UW library gateway using Neilson’s 10 heuristics, cognitive walkthrough of question 5-8 provided by the CNAG group, competitive analysis of U of Arizona

Report 2:
Discount usability, compile information into one report, intent of methods, findings, proposal justification, editing final report

Report 3:
Final editing and modifying report 1 methods, statistical analysis, lab based usability testing.

KJ Shipp

Report 1:
Heuristic evaluation of current UW library gateway using Norman’s 4 heuristics, cognitive walkthrough of questions 17-20 provided by the CNAG group, competitive analysis of University of Pennsylvania, editing final report

Report 2:
card sorting, developed interim prototype 2,


Report 3:
lab based usability testing, modified prototype 2

Melissa Thomas

Report 1:
Heuristic evaluation of current UW library gateway using Neilson’s 10 heuristics, cognitive walkthrough of questions 9-12 provided by the CNAG group, competitive analysis of the North Carolina State University and Dalhousie University library websites

Report 2:
Discount usability, task breakdown, methodology, and findings from discount usability

Report 3:
Keystroke level analysis, methods of keystroke level analysis and usability testing, task allocation

 

Appendix B - Summary of Team Learning

Learning of Methods

One of the major areas of learning within this project was identifying and using different methodologies to identify and solve usability problems. However, after having implemented these usability methods, it was found that there were many shortcomings to most of the methods. Some of the major findings are discussed at length below in an individual section for the methods.

Heuristic Analysis

Even though heuristics are widely used because they are so ‘intuitive’ it was found that applying these heuristics is anything but common sense. The expert user truly needs to be an expert to comprehend the scope of the problem and how this relates to the heuristic. For example, while "Aesthetic and minimalist design" may appear to be an easy heuristic to evaluate, the precise definition of what is minimalist is up for debate. Additionally, how much minimalism is good compared to the usability issues developed when too much minimalism is applied? These questions, and others like it, come back to haunt the design team when they attempt to apply such a ‘common sense’ method blithely.

There are also other drawbacks that were apparent when using heuristic analysis to evaluate the usability of the web pages. Firstly, the heuristics themselves identify possible areas of problems based on the method of definition. However, even though a consensus was reached prior to the evaluation as to the exact definition of each heuristic, it is still largely up to the expert user and evaluator to interpret what the heuristic means in relation to the web page.

Secondly, evaluating the heuristics in a group of individuals increased the number of ideas generated regarding one specific topic, however, this also had a tendency to polarise the design section into focusing on specific areas of interest that were mentioned early in the evaluation.

Lastly, while heuristic evaluation identifies possible problems with the web page, they provide no insight as to how to solve these problems. Identifying what to do after these problems have been identified is left entirely to the design team.

Design Walkthrough

It was learned that through a design walkthrough, some of the problems that users encounter solving real problems could be revealed. However this method of evaluation can be prone to errors in many locations. User testing for example is one of the areas where issues may occur. In attempting to ‘help’ the design team, each individual user will attempt to be on his/her best behaviour in attempting to solve the problem. This undesired side effect of overt observation may influence the results obtained by the user especially in timing critical issues. For example, a user will apologise and try to rationalise why they cannot find a link, instead of continuing on to try to find the link. And, oftentimes, when the user would have otherwise given up, they will continue looking until the desired link is found.

Another example where user testing can induce biases and errors in results is in the completion of surveys. During the final phase of design where the participants were filling out surveys, there is a noted bias in the evaluation of the gateway. This is likely due to the users attempting to be ‘nice’ or again, help the design team by encouraging them with their feedback. It is very difficult for these users to provide negative feedback when the individual who they are providing feedback to is hovering above them watching them fill in the survey.

Card Sorting

The card sorting method was discovered to be a quick method of developing a sense of what the user’s mental model may be. However, it was found that in the current study, that the results can differ dramatically in a small set of users. Thus, in order to properly apply this technique, it is likely that a larger set of users needs to be tested to ensure that the mental model of the greatest number of users is encapsulated in the implemented model.

This poses a difficulty for user centred design as the design process is already time limited and testing a large number of users in this fashion will likely take more time than is available. Thus it is incredibly difficult to encapsulate the mental model of all the users and one must evaluate these issues on a more heuristic level.

Statistical Analysis

In the study performed, it was found that there was very little significant difference for user performance in almost all of the cases. There was one case where the difference was significant, however this can be accounted for by one value which was eight times the mean value. This single value skews the entire set of results when the number of samples is small.

Thus, for accurate statistics which have significant differences, large numbers of samples are required. However, these are not readily available in a usability study due to time constraints and budget constraints since the main product usually developed depends on many other factors and usability is but a small facet. Thus, as with the card sorting, small inferences can be drawn from the use of statistics, however for the large majority of the time, it is difficult to gather enough data to detect significant differences in means.

Learning in regards to Teams and Team Dynamics

It was felt that working closely as a team with frequent meetings and progress reports of tasks help improve the team dynamics and team performance, as well as the quality of individual assignments. Unfortunately, when members of the group are unable to make meetings, work quality and quantity suffer as fewer people are towing the weight of many. It should be kept in mind for future group work that the attendance at meetings is crucial for both the divvying up of tasks as well as the reporting of results from previous assignments.

It was also learned that each of us may have different priorities and schedules in school life that co-ordinating tasks and meeting can be challenging at times. It is recommended that each of the group members allocates a period of time that is used solely for the purpose of this group work. This will make meetings with everyone more likely to happen. It is noted that each one of us may be absent for a particular meeting due to unexpected events, but it is essential to be warn the rest of the group of his/her actions.

For future group work, the research group has learned not to divide up tasks for a particular report and put it up back together. The result of that type of work was very poor as reflected by our first interim report evaluation. Also, it was felt that it was beneficial to work together at the same time, which allowed more exchange of information and better co-ordination of tasks.

 

Appendix C - Summary of Client Minutes

Date:
Time:
Present: Ian Dai, Phat Ha, General Leung, Melissa Thomas
Absent: KJ Shipp
Location: Dana Porter Library
Agenda: Introduce team members, discuss the scope of the project, find out what has been done in terms or testing thus far and find out what the requirements are for the gateway.

The scope of the project is to cover the main gateway page only, any comments on the underlying pages are welcome but they are not to be included in the testing process. Changes to the main gateway may affect the underlying pages in terms of maintaining consistency of fonts, colour and formatting between all pages. The client felt that we should feel free to make any changes we want no matter how drastic they are from the current site as long as they can be justified.

The current UW library gateway page was developed by in house graphics people and has not been tested at all.

The UW gateway page must provide links to Trellis, e-journals, e-data, e-text and possibly reference tools as well as any other points we determine to be used frequently. The gateway must accommodate varying user types such as providing short cuts for expert users.

The client will get back to us on frequency of use of the different links on the gateway page as well as the monitor resolution of the library computer stations. Client asked if it would be possible to be involved in future user testing and the possibility of the use of focus groups. Group to get back to client as the project moves along on this possibility.

Meeting adjorned.

 

Date: March 2, 2001
Time: 2:30 - 3:00
Present: Michelle Laing, General Leung, Phat Ha, Melissa Thomas
Absent: KJ Shipp, Ian Dai
Agenda: Discuss the work that was done for interim report 1 and show Michelle a copy of the first prototype, discuss what will be happening in the second phase of the project.

We showed Michelle a copy of our first interim report and walked through the report with her explaining what heuristics are and what cognitive walkthroughs are and what we found from these usability methods.

We discussed the pros and cons of the UW gateway page in comparison to the other university library pages based on our competitive analysis.

We showed Michelle the first version of our prototype and explained why we had changed the UW gateway to our version. We discussed some of the common findings between all of the groups on the current UW gateway page in terms of both good areas and areas for improvement.

We explained what we would be doing for further testing in the second phase; cardsorting and discount usability, and that users would now become involved in the testing. We explained the reasoning and process behind cardsorting and discount usability techniques.

We discussed some of the problems that we saw in our current prototype and ideas that we had to make improvements upon it.

Meeting adjorned

 

Date: March 20, 2001
Time: 12:30 – 1:30
Location: Dana Porter
Present: Phat Ha, General Leung, Melissa Thomas, KJ Shipp
Absent: Ian Dai
Agenda: Review comments from report 2, perform a trial run of the lab based usability testing that will occur on March 22 and discuss how the testing will be carries out.

Michelle had read the second report and had some comments on our current prototype. "View Patron Information" menu title was a bit wordy, why not just use Patron Information. We decided there were no real reasons for this and would take her recommendation. The "more" under course information takes you to the same information that has links provided for it on the gateway and there is no real for this to be here. We agreed and will change that, it had been there originally as it was present in all of the other menu items. "View Patron Info" could be shortened to just "Patron Info". The link to "overdue books" is somewhat negative; if it could changed to be more positive. The links at the bottom of the page on the menu bar "site map" and "search" take you to the same information if they could be combined into one it would reduce the amount of information presented. Different font sizes are used in the different columns of the page and the middle column items are difficult to read as they are in small font.

We performed a trial run of the usability testing to ensure that our gateway page was correctly working. Michelle was able to find all of the links necessary to answer the questions. She noted that some of the ways she was accessing information she thought many students would not know about. We had a link to a text version on our page, which was unnecessary, as our prototype is a text version. The navigation bar is squished together and difficult to read. Michelle liked the quick search for Trellis and the UW site at the top of the gateway page. However the Trellis search should be on the top as this is the library page.

We explained that would be doing similar testing on users on March 22. The users would primarily be people we know and likely be a lot of engineering and applied health studies students. The testing will allow us to see how users of a variety of levels of experience will perform using our gateway.

We told Michelle that our final report was to be done for March 27 and we, along with all of the other groups would be presenting our findings and final prototypes to the class on that day. We thought that Professor MacGregor would be contacting the CNAG group to invite them to see the presentations.

Meeting Adjorned.

 

Appendix D - Heuristic Analysis

Heuristics Used in Analysis

The following heuristics were used in the research group’s analysis of the gateway. These were selected due to the familiarity the research group has with these methods. As well, these sources are widely quoted in various other usability texts.

Norman’s four principles:

  1. provide a good conceptual model
  2. make things visible
  3. provide good mappings
  4. provide appropriate feedback

Nielson’s ten usability heuristics:

  1. Visibility of system status
  2. Match between system and the real world
  3. User control and freedom
  4. Consistency and standards
  5. Error prevention
  6. Recognition rather than recall
  7. Flexibility and efficiency of use
  8. Aesthetic and minimalist design
  9. Help users recognize, diagnose, and recover from errors
  10. Help and documentation

Interpretation of Heuristics

A standard method of interpretation of each heuristic was required when the evaluation was performed. Below is summarised a brief description of how each of these heuristics were conceptualised.

Conceptual Model

This heuristic addresses the difference between the users conceptual model, and the physical model; how the user thinks it works versus how the object actually functions. Areas where the conceptual model does not match the physical model should be noted through this heuristic.

Mapping

The mapping heuristic identifies affordances that the gateway provides, or devices that the design provides to tell the user how to interact with the interface. This heuristic tends to reveal similar information as the above heuristic, however not surprisingly since mapping is a subset of how the user perceives the design.

Visibility

This heuristic identifies what is unclear about the interface, as well as what is well presented. The clutter or organisation of the web site should be identified through the results of this heuristic.

Feedback

The amount of information the user receives about the system status is evaluated under this heuristic. It evaluates whether the information is presented or lacking, in addition, it also assess the appropriateness of the information presented when it is presented.

Consistency and standards

The interface is evaluated for the uniformity of language and presentation through this heuristic. Objects that do not fit uniformly are identified, and situations or actions that are different from the norm are also singled out.

Recognition rather than recall

According to Nielsen:

"The user should not have to remember information from one part of the dialogue to another." Nielsen’s website useit.com

Thus, this heuristic identifies any portion of the interface where the user needs to remember information rather than having it directly presented to them.

Flexibility and efficiency of use

This heuristic analyses the accelerators implemented to speed up the interaction for the expert user. Additionally, how quickly the user can access the available features is evaluated. <4>Aesthetic and minimalist design This heuristic identifies irrelevant or rarely needed information, as well as the complexity and the initial appearance of the interface.

Help and Documentation

This heuristic should evaluate the accessibility of help and documentation from the main gateway page.

Findings

The results of each heuristic evaluation is presented below underneath its appropriate heading.

Conceptual Model

On the UW library page, there is a drop down list in the top right hand corner that only uses descriptions that are useful for expert users (e.g. TUGdoc, ILL). Additionally, the drop down list is inconsistent with the rest of the page; other lists are presented as floating graphics that appear when the appropriate menu item is scrolled over, and new menus appear. The user then needs to click on this item to proceed, however with the drop down box, the user must select their desired item and then press the ‘Go’ button.

The main access menu down the left hand column (e.g. "Find It" and "Get It") appear to be links, however they are not and clicking on them produces no action.

A menu bar appears at the top of the screen which again is inconsistent with the rest of the page. Firstly, the buttons do not appear to be buttons at all and are slightly obscured by the banner of the library. The two meet and seem to join together and the links are lost within this banner. Additionally, these buttons have no descriptions attached to them, unlike the main access menu which pops up an additional window.

Mapping

The link to the UW main page from UW logo is not obvious unless one mouses over the logo. This is not conceptually clear and does not provide good affordances to suggest its use. Another area that may prove confusing is the access menu which appear to be buttons. If one clicks on them, nothing will happen, when one expects to be linked elsewhere. As well, accidentally clicking on the blank space in the floating list could actually link to another page.

Visibility

The one large background graphic distracts from the text of the page, as well, obscures completely the link to the text version of the page. The links at the bottom of the banner are not obvious as they do not protrude, or in any way make themselves more noticeable than the rest of the banner.

Some good visibility issues with the site include good colour contrast in the floating lists. As well, the main navigation buttons down the left hand side of the page change colour and an arrow appears pointing to the floating list when the user scrolls over these links. Within the lists themselves, there is high contrast between the text and the background to facilitate reading of the text. As well, resizing the browser window does not effect the formatting/readability of the page.

Feedback

Popup windows appear when the user mouses over the menus on the left. The system does not in any way show that this new pop up menu is to be clicked on, it does show an arrow pointing to the right indicating something significant is supposed to happen. On the plus side, the window pop up interface does appear immediately when scrolled over, indicating that the window has popped up due to the action of scrolling over the link. Contributing to the confusion regarding the pop-up menu is the amount of white space between the menu heading and its associated pop-up menu, that is, there is no visual link between the two related interface elements. Dynamic menu systems should bear some resemblance to commonly adopted styles such as those found in the Windows 9x operating systems.

Consistency and Standards

The web page provides some terms that may be foreign to the novice user of the system. For example, TRELLIS, and TUGdoc, which both appear on the first page of the interface have no attached explanations. Additionally, Find It and Get It may mean different things to different users, and it is not obvious or evident what is under these menus until the user actively scrolls over them. It is not clear if the menu heading has any bearing on the function of the associated menu item. For example, if an item is found via Find It, are users able to obtain the item, that is, get the item.

Additionally, the same pop up menu does not appear for other links on the page such as the buttons located at the top of the page. When the user scrolls over these menus, the status bar at the bottom of the browser as well, the cursor pointer changes shape. These are inconsistent with the mouse over menus.

Recognition rather than recall

Not all the links that are available from the University Library webpage are instantly available when the page is first loaded. The user must mouse over the appropriate selection, and another side menu will appear. This forces the user to remember what appears in which column, possibly confusing some of the users. On the other hand, the user will have to recognise what the menu items stand for, discriminate it from other menu items according to what they want to accomplish, and then select and action.

Although directed towards the expert user, the Library Forms menu item descriptions are entirely abbreviations and will not foster acceptance from novice users.

Flexibility and efficiency of use

The accelerators that are put in place on the gateway include the link buttons across the top of the page, as well as the drop down menu system on the top right hand side of the page. These accelerators allow the advanced user to access these functions quickly, however some of the links that are possibly more commonly used (e.g. TRELLIS book search) are not available from the gateway.

Aesthetic and minimalist design

Rarely required information, such as all the items underneath the help menu, are displayed on the main gateway. Aside from a possible sentimental value the existing background does nothing to improve usability, increases loading time, and forces an offset of the contents of the page in order to minimise conflicts with poor contrast.

Help and documentation

The help and documentation is available directly from the main page. Through this, many different varieties of help are available: research workshops, and online tutorials can all be accessed in addition to general help about TRELLIS. However, no direct feedback on the page is available. The contextual pop-up help messages that became visible on mouse-over of a menu heading were simply reiterations of the menu heading descriptor, that is, it does not offer any new information.

 

Appendix E – Cognitive Walkthrough

List of 20 Client Supplied Questions
  1. How would you find if the library has Margaret Atwood’s Alias Grace?
  2. Is the University Map and Design Library open on weekends?
  3. I need to contact the librarian for my department – how do I find his/her phone or e-mail?
  4. Does the Library have any electronic dictionaries?
  5. Where can I find electronic maps?
  6. Where do I find the URL for Yahoo?
  7. Where can I find a database in which to locate articles on Anthropology subjects?
  8. Where can I find information on how to cite web sites?
  9. Can I read an article of a journal without coming to the library?
  10. Guelph Library has the book I need. How do I have it brought to UW?
  11. How can I discover if any materials have been placed on Reserve for my courses?
  12. How can I discover when my UW library books are due?
  13. I am visually impaired – does the library offer any special services for me?
  14. Where can I find the exam timetables?
  15. How can I find out if my course textbook is in stock at the UW bookstore?
  16. Where can I find instructions about connecting from home?
  17. I need to find a book that is not available in our catalogue – how can I get acopy from a remote library?
  18. I need to find a book that is not available at the University of Guelph?
  19. I am a new undergraduate student – I need to do research on a topic – where do I start?
  20. What library orientation sessions are being offered for graduate students this term?

Results of Cognitive Walkthrough

User Testing Phase 1

Presented below are the answers to the ten questions as given by two users. The first column shows the link from the gateway that was selected. The second column indicates the link that the user selected .
  User 1 User 2
  First page link selected Underlying link selected Time First page link selected Underlying link selected Time
1 About the library Hours Quick About the library Hours Quick
2 About the library Staff Administration Slow About the library Staff Administration Quick
3 E-Text   Slow Reference tools   Medium
4 E-Data   Quick Journal Indexes   Quick
5 Borrowing Materials From Guelph Quick Borrowing material From Guelph Quick
6 Course Information Reserves Quick Course information Reserves Quick
7 Renewals   Quick Borrowing Materials From waterloo Medium
8

User gave up

 Course information Booklook Quick
9 About the library Services for Quick About the library Events Medium
10 About the library Services for Medium About the library Guide to the libraries medium

Conclusions

Both of the users knew which links to go to quickly to find out the hours of the map and design library. Both users were a little confused at first because they did not know what the map and design library and that it existed at Waterloo. The link for finding information about the libraries and the hours was intuitive to the users and therefore there is no need to change this aspect of the current design.

Both of the users were able to figure out the links required to find out how to contact the staff person in their department. User 1 was slow as he did not know the service existed. The design will remain the same for this feature, as both of the users were able to figure out the necessary links.

Only one of the participants was able to find the link to the dictionaries and it took some time to find. The other participant thought that it might be contained under E-Data after scanning all of the other menu items. The title reference tools does not provide a clear conceptual model of what is contained in the link, however the objects listed therein are reference tools, and when questioned after the testing what a reference tool was, the users invariably cited dictionary.

Neither of the subjects selected the research tool by subject link. Instead E-Data and Journal Indexes which both could be possible methods of searching for information on a specific subject but was not the intended link from the question. Journal indexes is a common search method among users and is more familiar to them which may be why the user selected that link. The design was changed our to eliminate the "research tool" from the link title so that is only "by subject". This was done so that the word subject would be closer to the left and more salient when scanning the list of links.

Both of the users went quickly to the Borrowing Materials link and then right to From Guelph. This indicates that links are intuitive. One could also select the Library Forms link which could also be used for this function. It is more likely to be used by more expert users who a familiar with the acronyms used for the form names. The Borrowing Materials was left the same but some changes were made to the Library Forms to make it more visible and show what was located in the link.

Both of the users took the same route to find information about course reserves and were quick in taking that route. This feature worked well so it was left the same. Course reserve information can also be found out through Trellis which would be a more direct for expert users.

The users took different routes to find out when their library books are due. The pages they went to gave them instructions for renewing material which is the same as what is required for finding out due dates. The links selected are indirect methods to finding out the information. In an attempt to fix this problem the renewal renewals link was removed so that there was only the link to "View Patron Info". Additionally, a description was placed under the link of what information was contained in that link. This should provide the user with more information.

In trying to find out if the course text is in stock at the UW bookstore one of the users gave up. The other user was able to find this information quickly though under course reserves. To try and improve this the description under "Course Information" link was changed to include textbooks.

The users selected different methods to see what orientation sessions were being offered to graduate students. They both got the first link correct but from there one user selected "services" and the other user wasn’t quite sure what to select and settled on "Events". The "services" link is a more direct route than the "events" link. To try and improve this a "Services" link was created right on the first page with a description of what was contained inside the link.

Both of the participants had trouble with finding information about connecting from home. Both went to "About the Libraries" and then took different paths from there, one user selected "Services For" and the other user selected "Guide to the Libraries". This information is actually contained in the help menu which is somewhat hidden at the bottom of our prototype with no description. To address this a visible Help link was placed on the gateway with the other links to make it more visible. The help function is still available in the menu at the bottom, as this is meant to be consistent within all of the UW library pages.

The cognitive walkthrough identified a number of problems that the users had in finding certain items within the new gateway page. Finding patron information was an area that created difficulties and was addressed by removing the Renewals link and providing a description of the information contained in the View Patron Information link. Finding electronic dictionaries was a problem in which the research group has no solution to at this point. A quick link to the Trellis page was added to improve the use for expert users. The prototype after this phase of design can be found in the following section.

Heuristic Evaluation 2:

Minimal problems were observed feedback while interacting with the gateway and with compliance with standard internet methodologies for web publishing. Help and documentation were easily recognisable and can be accessed directly from the gateway.

The entire top portion of the page should be visually distinct from the bottom portion. This will separate the content, (i.e. the links to the underlying pages, and the structural elements, such as, the title and UW logo). With the exception of the "Link Bar" at the bottom of the page all visible textual links should lead to different pages. This will require that the centre column of links be reformatted so that either the headings are links and the descriptions are not or vice versa. Alternatively, since the descriptions are also links in the underlying page they can be reformatted to link directly with the appropriate underlying page. This will eliminate the confusion as to which link should be selected.

More work is needed to accommodate all web elements into a standard 800x600 resolution monitor in order to eliminate the need for scrolling. This may require the resizing of fonts, decreasing space between paragraphs, and rearrangement of menu headings. Expert functions must be incorporated directly into the gateway, as opposed to linking to a page listing the expert functions, and should be recognisable by current expert users. Otherwise, the effort required to access the functions through the expert function would be no different than the non-expert method. An option would be to emulate the existing "Library Forms" expert function as found on the original gateway. Likewise, access to the Text Version of the gateway should be implemented as this is a design requirement.

Although the use of graphics is minimal, there is little evidence, from the two prior heuristic evaluations and one design walkthrough with two users, showing that incorporating graphics will increase usability. That said, graphical elements may be used in the future to highlight the top portion of the gateway to separate it from the content portion and to satisfy aesthetic values of the client.

User Testing Iteration 2:

Presented below is the answers to the ten questions as given by two users. The first column shows the link from the gateway that was selected. The second column indicates the link that the user selected.

  User 1 User 2
  First page link Underlying link Time First page link Underlying link Time
1 About the library Hours Quick About the library Hours Quick
2 About the library Staff Administration Quick About the Library Staff Administration Quick
3 No answer   Slow No Answer   Slow
4 Journal Index   Quick Library Catalogue   Medium
5 Borrowing Materials From Guelph Quick Borrowing material From Guelph Quick
6 Course Info Reserves Quick Course information Reserves Quick
7 View Patron Information   Quick View Patron Information   Quick
8 Course Information Textbooks Medium Course Information Textbooks Medium
9 About the library News/Events Medium About the library News/Events Medium
10 Help Connect From Home Medium Help Connect From Home Quick

Conclusions

From the second user testing iteration, it was noted that the users have consistently shown trouble with identifying the services for graduate students and finding a link to an electronic dictionary. The majority of users point to News and Events under About the Library to look for the services and have given up looking for the dictionary link.

Most users believe the small text underneath is a quick link directly to the web page, whereas it points to the same link as the headings. The users have commented on the general easy accessibility of the proposed gateway page. Some expressed concerns over the plain nature of the heading, graphics are suggested for the next prototype.

In the final prototype of the discount usability method iteration, it was observed that the time to complete the task and the accuracy in answering the questions were improved over previous prototypes. However, from the general comments received during testing, prototype 3 is still lacking in graphical design. To increase the visibility of the headings and the visual appeal of the web page, icons are intended to replace major headings in the next iteration of design.

As other minor changes, the sub menu items under the headings which are currently just one link, are going to be broken into separate links that directly link to underlying pages. This will increase the efficiency of the gateway page and make it faster access for experienced users. The Trellis link will incorporate an icon and text, and will be displayed on top of the gateway for instant access to the UW library catalogue.

UW Library Gateway Prototype 1

Figure 3: UW Library Gateway Prototype 1

UW Library Gateway Prototype 2

Figure 4: UW Library Gateway Prototype 2

UW Library Gateway Prototype 3

Figure 5: UW Library Gateway Prototype 3

 


Appendix F - Hierarchical Task Analysis

Use Library Forms

Analysis

This type of list is different from the floating lists that are used on the remainder of the page in that the user selects where they want to go by selecting the item from the floating list. This feature has an additional step of clicking on the ‘GO’ button after selecting the list. A different type of list that is more consistent with the rest of the gateway should be used.

Use Find It

Analysis

A potential difficulty with this feature, and each button that activates a floating list, is that the user has to move directly from the button to the list. If they accidentally move the pointer across another button, then the list changes.

Use What’s New List

Plan 0
DO
1.0
2.0
END 		Plan 1
DO
1.1
1.2
1.3
END
Do only one of 1.1-1.3

Analysis

There are three ways to get the ‘What’s new’ floating list to appear. The problem is that the location of the list is not clearly connected to any of the ways of making the list appear. The button at the top looks like a link and should act like a link, but instead it just reloads the page. Resetting the list by moving the pointer over the bottom of the page was discovered by random searching. There should be a more intuitive way of retrieving the ‘What’s new’ list, or the list should be available at all times.

Summary

From the task analysis it can be seen that getting to any of the available library pages can be done with a minimum of steps, but the methods of getting the lists of links is inconsistent and in some cases confusing. A more consistent and clear method should be used to link the main title and the sub-titles associated with it.

 

Appendix G - Competitive Analysis

University of Arizona

The conceptual model provided by the Arizona State University is superior in several ways over the conceptual model of the University of Waterloo web site. Firstly, the icons at Arizona State have more appropriate and verbose descriptions than do the sparsely worded Find It and Get It icons do on the UW website. Additionally, the ‘mouseover’ menus provided on the UW website are more confusing and not as standard as the Arizona State graphical link paradigm.

Mappings are generally good in both cases. The ‘How to Find’ drop down on the left of the Arizona site seems to indicate a help menu of sorts, however it provides the links to items which you need help "trying to find". As previously mentioned, the increased number of words (e.g. "Catalogs of Books and More") make the mappings on the Arizona State Web site more intuitive and natural, however, for an intermediate to advanced user, the mappings on the UW website are manageable. The visibility of the Arizona state website is much poorer than the UW website. For all the intuitiveness that the extra text adds to the links, there is an increase in clutter associated with this increase in clarity. A region where the Arizona state website may surpass the UW website is in that it has no background graphic that distracts the user’s attention away from the task at hand.

Both sites provide the same standard of feedback, both implementing the same tricks of showing the site of the link on the status bar, as well as the cursor changing and a popup window displaying what the name of the link under the arrow is entitled. Both sites require that the user memorise very few things and most of the information is provided directly on page.

However, as far as documentation, there are a noticeable number of acronyms utilised on the UW website as opposed to the Arizona State website. This may be a case where introducing an acronym may be counterproductive to the understanding, and therefore, the usage of the underlying page by a user because they do not understand the acronym. Both interfaces are very easy to use, although a novice mouse user may have difficulty with the concept of ‘mouseover’.

There should be a simple mechanism that follows a graphical link paradigm such as that shown on the Arizona State library web page. Background wallpapers or graphics that make the site look cluttered should be removed. The Help and Documentation is provided in a much better and easier to access position on the UW page than on this competitor’s page.

University of Washington

The University of Washington’s Library gateway is very similar to that of the Text Version of the University of Waterloo with respect to its layout and menu headings and elements. It does not have expert functions on the main site and considerable scrolling is needed to view its entire contents. Contrast is poor and variable because the background is fixed while the text flows in parallel to the scroll bar. Many of the underlying links shared a common navigation and bordering system and layout, although, the main menu items are redundant in these pages. Users are able to compare all the menu elements from one menu to another without having to remember the menu contents when accessing a new menu. One main difference is the use of image-based links rather than the commonly adopted blue underlined text – this is not the case on the underlying pages. Because of its minimal graphical content the page is able to load quickly. Many of the page elements, such as, menu groups and menu headings, are misaligned and do not share a common imaginary border.

NCSU

The NCSU library main gateway page is set up with a column of menus on the left hand side of the screen. All of the menu items are displayed at all times. The menu is divided up into three sections; services, about the library and site help. It is difficult to tell which menu item you are about to select, as there is no colour change or anything to indicate which item you are over and the items are quite close to each other. The page has very minimal graphics on it, it is mainly words on the page. The library catalogue search is located right on the main page and requires no links. The centre section of the page has a section on databases, e-journals and websites which each have a brief description of what is contained in the underlying page. The fact that a description is needed indicates that there is no clear mapping of those functions. There is a "spotlight on" section that has a few current events and items of interest highlighted.

Both the UW and the NCSU website provide a good conceptual model of how the links function. The NCSU catalogue has a weak conceptual model as all of the items require you to go to an underlying to use the function but the catalogue search is right on the gateway page, which is not consistent with the rest of the functions. The visibility of the NCSU gateway is much clearer than that of the UW website as there are no graphics in the background. NCSU uses a text based gateway page and all of the menus are visible at all times.  There is not a clear mapping between the NCSU library catalogue and the drop menu that is the catalogue. The NCSU gateway page does not provide feedback as to which item the mouse is on whereas on the UW site the menu changes colour making it apparent to the user what is selected.   Both of the gateways have all of the information present on the screen so that the users do not need to recall much information, . NCSU provides different levels of searches for more expert users who know where they want to go. UW provides a similar feature for more expert users with the drop down menu.  The NCSU gateway has a very simple design with the page divided up into two main columns and very minimal graphics whereas the UW gateway is graphics based which can be a little distracting to the user. The NCSU gateway provides some help links in the site help menu and the "ask a librarian" function but is less visible than compared with the UW page.

Yale University

The UW site and the Yale site both have different models for their design. Yale’s model relies on simplicity and lacks in textual information. UW’s page has more features and functions. The Yale model seems better suited for first time users and UW’s model is superior for experienced users. UW’s gateway offers a clearer mapping than the Yale library main page. UW’s page also provides faster and more efficient access for experienced users of the library pages.  Although UW’s page has fairly good visibility properties, Yale’s gateway is a fresh and visually appealing design that uses graphics over text. Yale’s page is simple, everything is visible and clear, and users of this page would have an easy time identifying the links available. UW’s page relies on the mouse over option to present the necessary navigation information.  UW’s gateway provides superior feedback as there is hardly any feedback information from the Yale’s main page. This is the one area Yale’s gateway lacks. .

Yale’s simple presentation does not rely on user’s recall, it is clear enough to recognise the main links immediately upon opening of the page. UW’s floating lists are clearly displayed and are easily accessed so that users do not need to recall the procedures from the time before. Although the dropdown list box option requires users’ to have previous experience.  UW’s gateway proves to be more flexible and efficient to use, with more links from floating lists and direct dropdown list access, it is easier to access information without going through several layers of intermediate pages. Yale library’s main page provides much better aesthetic design by using visually appealing graphics. It also has significantly less information than the UW’s page. UW’s gateway provides better help and documentation, since Yale’s page does not have any help links at all.

University of Pennsylvania

The links are clear if one realises that everything is a link. The main page can be seen without any horizontal scrolling, so it is easy to see everything at one time. Mapping on the UPenn site is done by grouping the links to simply finding the right link. The UW site uses more rigid grouping where only one group of links can be seen at one time.  The links do not have a standard format, which can make them more difficult to determine what is a link and what isn’t, but almost every word on the page is a link. It is visually nice to be able to see every possible link from one page. The page is also visually appealing through the use of colours and style to signify different groups of links.

The only feedback available to the user from the UPenn site is the mouse changing when the pointer is over a link, and being taken to a new page when a link is selected.  The entire UPenn library site is consistent; the top and bottom button bars appear on every page with only a few exceptions. This helps make the buttons easy to recognize. The links are not standard format on the main page, but it is easy to determine that the titles are links.  The UPenn site is very easy to use, all possible links can be seen at once, and standard links are seen on most pages. There is a search window available on the gateway to help expert users find information quickly. The UW website makes use of the changing floating list to provide a lot of links on one page, the UPenn site provides a lot of links, but they are all shown at once. This has the advantage of being able to see everything at once without having to ‘mouseover’ buttons to see new lists.

The UPenn site design is kept simple by the limited use in colours, the UW site is kept simple by limiting the amount of information seen at one time.  Compared to the help available on the UW library gateway, the UPenn gateway is simpler. On the UW site you have to guess what type of help you need, when you select help from the UPenn library site it takes you to a page that lists the area’s of help available and provides a description of each.

Dalhousie University

The Dalhousie gateway does not provide a clear conceptual model of how the ‘mouseover’ menus function whereas the UW page is clear. The menu and sub menus in the UW site are more clearly linked together than Dalhousie’. Both of the pages have some ambiguous links such as the library names and catalogues, which may not be understood by the entire user population. The UW site ‘mouseover’ menus do not cover up any of the other menus although you can only see one sub menu at a time the main menus are always available. On the Dalhousie site the drop down menu covers the rest of the menu items in the right most column. Dalhousie’s gateway has good feedback as to what menu you are selecting as it changes colour, which is the same method the UW site uses. Both sites allow you to access an alternative version of the web page, Dalhousie has their old page available and UW has the text version available. Dalhousie has a quick links drop menu for more expert users, as does UW gateway. Both pages offer the user different ways to access the same information. The Dalhousie site is organized nicely into different sections but the page is quite dark. The graphics used requires a multimedia adapter which all users may not have on their home computer and takes longer to load. The UW gateway also has a graphic which increases the loading time of the page. The UW page is brighter and more pleasing to look at. The help menu on the UW site much more clear to the user and provides more help options. The Dalhousie help item is hidden in the ‘How Do I?’ section which may make it more difficult for the user to locate.

 

Appendix H – Phase 3 Results

Results for the Keystroke Level Analysis

Where would you go to find general help with a research topic in your department?

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (Information by Subject) 1.10
3 Click on link 0.2
Total   2.65

Novice User

Step Operator Time
1 Mentally prepare 1.35
2 Scan top bar (6 chunks) 6.0
3 Scan left side links (7 chunks) 7.0
  Point to link (Information by Subject) 1.10
  Click on link 0.2
Total   15.65

Additional Routes to same task, #1

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (Staff) 1.10
3 Click on link 0.2
Total   2.65

Route #2

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (Help) 1.10
3 Click on link 0.2
4 Point to link (research guides) 1.10
5 Click on link 0.2
Total   3.95

How do you get an article from a journal that is available at WLU?

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (WLU) 1.10
3 Clink on link 0.2
Total   2.65

Time for Novice User

Step Operator Time
1 Mentally prepare 1.35
2 Scan top bar (6 chunks) 6.0
3 Scan menu titles (3 chunks) 3.0
4 Scan menu items (3 chunks) 3.0
5 Point to link (WLU) 1.10
6 Clink on link 0.2
Total   14.65

Alternative Routes

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (Borrowing Materials) 1.10
3 Click on link 0.20
4 Point to link (From Guelph/WLU) 1.10
5 Click on link 0.2
Total   3.95

Can alumni borrow books from UW?

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (Borrowing from UW) 1.10
3 Click on link 0.20
Total   2.65

Novice User

Step Operator Time
1 Mentally prepare 1.35
2 Scan top mar (6 chunks) 6.0
3 Scan menu titles (3 chuncks) 3.0
2 Point to link (Borrowing from UW) 1.10
3 Click on link 0.20
Total   11.65

Alternative Routes, #1

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (services) 1.10
3 Click on link 0.20
4 Point to link (Alumni) 1.10
5 Click on link 0.20
Total   3.95

What are the hours for the University Archives?

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (Hours/Locations) 1.10
3 Click on link 0.20
Total   2.65

Novice User

Step Operator Time
1 Mentally prepare 1.35
2 Scan top bar (6 chunks) 6.0
3 Scan menu titles (2 chunks) 2.0
4 Scan menu items (3 chunks) 3.0
5 Point to link (Hours/Locations) 1.10
6 Click on link 0.20
Total   13.65

What books are available at the Kitchener Public Library?

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (library catalogues) 1.10
3 Click on link 0.20
Total   2.65

Novice

Step Operator Time
1 Mentally prepare 1.35
2 Scan top bar (6 chunks) 6.0
3 Scan menu titles (1 chunk) 1.0
4 Scan menu items (9 chunks) 9.0
5 Point to link (library catalogues) 1.10
6 Click on link 0.20
Total   18.65

Alternative Routes

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (local libraries) 1.10
3 Click on link 0.20
4 Mentally prepare (read menu) 1.35
5 Point to link (KPL) 1.10
6 Click on link 0.20
Total   5.30

Alternative Routes # 2

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (Borrowing Materials or more) 1.10
3 Click on link 0.20
4 Mentally prepare (scan list) 1.35
5 Point to link (other libraries) 1.10
6 Click on link 0.20
Total   5.30

Does the library have any electronic dictionaries?

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (reference tools) 1.10
3 Click on link 0.20
Total   2.65

Novice User

Step Operator Time
1 Mentally prepare 1.35
2 Scan top bar (6 chunks) 6.0
3 Scan menu titles (1 chunk) 1.0
4 Scan menu items (8 chunks) 8.0
5 Point to link (reference tools) 1.10
6 Click on link 0.20
Total   17.65

Does the UW library have a copy of Andrew Pyper’s Lost Girls?

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (Trellis UW’s Catalogue) 1.10
3 Click on link 0.20
Total   2.65

Novice User

Step Operator Time
1 Mentally prepare 1.35
2 Scan top bar (3 chunks) 3.0
3 Point to link (Trellis UW’s Catalogue) 1.10
4 Click on link 0.20
Total   5.65

Alternative Routes

Step Operator Time
1 Mentally prepare 1.35
2 Point to trellis drop down menu 1.10
3 Click on arrow 0.20
4 Point to author 1.10
5 Click on link 0.20
6 Point to menu bar 1.10
7 Click on menu bar 0.20
8 Type title 2.0
9 Hit enter or move and click on search 0.20 or 1.30
Total   7.45 or 8.55

Where can you find instruction for connecting from home?

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (connect from home) 1.10
3 Click on link 0.20
Total   2.35

Novice

Step Operator Time
1 Mentally prepare 1.35
2 Scan top bar (6 chunks) 6.0
3 Scan menu titles (6 chunks) 6.0
4 Scan menu items (2 chunks) 2.0
5 Point to link (connect from home) 1.10
6 Click on link 0.20
Total   16.35

Alternative Routes #1

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (Help) 1.10
3 Click on link 0.20
4 Point to link (connecting from home) 1.10
5 Click on link 0.20
Total   3.95

Where would you go to renew books online?

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (renewals) 1.10
3 Click on link 0.20
Total   2.65

Novice User

Step Operator Time
1 Mentally prepare 1.35
2 Scan top bar (6 chunks) 6.0
3 Scan menu titles (5 chunks) 5.0
4 Scan menu items (1 chunk) 1.0
5 Point to link (renewals) 1.10
6 Click on link 0.20
Total   14.65

What is the contact information for the Liaison Librarian assigned to your home department?

Step Operator Time
1 Mentally prepare 1.35
2 Point to link (staff) 1.10
3 Click on link 0.20
Total   1.65

Novice

Step Operator Time
1 Mentally prepare 1.35
2 Scan top bar (6 chunks) 6.0
3 Scan menu titles (2 chunks) 2.0
4 Scan menu items (4 chunks) 4.0
2 Point to link (staff) 1.10
3 Click on link 0.20
Total   14.65

 

Appendix I – ORE Form

UNIVERSITY OF WATERLOO

OFFICE OF RESEARCH ETHICS
APPLICATION FOR ETHICS REVIEW OF RESEARCH INVOLVING HUMAN PARTICIPANTS IN UNDERGRADUATE COURSE PROJECTS

  1. Title of Project: Usability Testing of UW Library Gateway Web Page

  2. Faculty Supervisor(s) Department Ext: e-mail:

    Carolyn MacGregor Systems Design Eng 2897 cgmacgre@engmail

  3. Student Investigator(s) Department e-mail: Local Telephone Number:

    SD 348 Winter 2001 students - see attached list

  4. Level of Project: Undergraduate Course Specify course and number: ___SD 348_____

  5. Indicate the anticipated commencement date for this project: __March 22, 2001___

    Indicate the anticipated completion date for this project: __March 30, 2001____

  6. Purpose and Rationale for Proposed Research

    Briefly describe the purpose (objectives) and rationale of the proposed project and include any hypothesis(es)/research questions to be investigated.

    We have been working on the redesign of the UW Library Gateway page in conjunction with the UW Library Community Needs Assessment Group (CNAG). The members of CNAG serve as clients for the 6 teams of students (5-6 students per team). All teams are working on the same assignment - the redesign of the UW Library Gateway page through the use of user-centred design methods. The methods and redesign activities to date have involved the students and the CNAG members. As the final phase of the design projects, the students will be carrying out formal "lab-based" usability testing of their final designs and the current UW Gateway page.

    The project does not involve a research question per se. The overall objective of the usability testing is to have the students experience a more formal method of evaluating their designs. The intention of the project is to generate recommendations for alternative designs for the UW Gateway page that can then be more rigorously tested by CNAG.

  7. Methodology/Procedures

    1. Which of the following procedures will be used? Provide a copy of all materials to be used in this study.

      [ ] Survey(s) or questionnaire(s) (mail-back) Are they standardized? All [ ] Some [ ] None [ ]

      [ ] Survey(s) or questionnaire(s) (in person) Are they standardized? All [ ] Some [ ] None [ ]

      [X ] Computer-administered task(s) or survey(s) Are they standardized? All [ ] Some [ ] None [ X ]

      [ X ] Interview(s) (in person)

      [ ] Interview(s) (by telephone)

      [ ] Focus group(s)

      [ ] Audiotaping

      [ ] Videotaping

      [ ] Invasive physiological measurement (e.g. venipuncture, muscle biopsies, catheter insertions, etc.)

      [ ] Non-invasive physiological measurement (e.g. exercise, heart rate, blood pressure, electromyography, muscle stimulation, balance/movement, force exertion, CO2 or altered O2 breathing, lower body negative pressure, etc.)

      [ ] Unobtrusive observations

      [ ] Analysis of secondary data set (no involvement with human participants)

      [ ] Analysis of human tissue, body fluids, etc. only

      Other (specify)

    2. Provide a brief, sequential description of the procedures to be used in this study.

      All users will be asked to read and sign the Information/Consent Letter. See Appendix A.

      All users will be asked to fill out a brief Background Questionnaire concerning their familiarity with the main UW libraries and the UW Library website (e.g. frequency of visits) and web pages in general. See Appendix B.

      Test monitors will read from a set script and each user will be asked to perform a set of tasks related to library activities using either the current UW Library Gateway Page or the redesigned Gateway page. Users will be reminded that the objective of the study is to test the usability of the design (and not to test the skills of the user). See Apprendix C.

      Test monitors will time how long it takes the users to complete the tasks, and will record when users have encounter difficulties in completing tasks (e.g. select a link that does not lead to the appropriate resource).

      See Appendix D.

      Participants will be asked to fill out a Usability Questionnaire once the tasks have been completed.

      See Appendix E.

      All users will be given a feedback letter. See Appendix F.

  8. Participants Involved in the Study
    1. Indicate who will be recruited as potential participants in this study.

      UW Participants:

      [ X ] Undergraduate students

      [ X ] Graduate students

      [ ] Faculty and/or staff

      Non-UW Participants:

      [ ] Children

      [ ] Adolescents

      [ ] Adults

      [ ] Seniors

      [ ] Persons in Institutional Settings (eg. Nursing Homes, Correctional Facilities, etc.)

      Other (specify) _________________________________________

    2. Describe the potential participants in this study including group affiliation, gender, age range and any other special characteristics. If only one gender is to be recruited, provide a justification for this.

      For the purpose of this design project, participants can be anyone who is a current UW student. In order to test the robustness of their designs, teams will be encouraged to recruit a range of participants (e.g. mix of males and females, undergrads and grads, different disciplines). The only requirement is that the participant has used web pages before.

    3. How many participants are expected to be involved in this study? 6 design groups X 6-10 participants

  9. Recruitment Process and Study Location

    1. From what source(s) will the potential participants be recruited?

      [ X ] UW undergraduate and/or graduate classes

      [ ] UW Psychology Research Experiences Group

      [ ] Other UW sources (specify) _______________________

      [ ] School Boards (not including local school boards)

      [ ] Kitchener-Waterloo Community

      [ ] Agencies

      [ ] Businesses, Industries

      [ ] Health care settings, nursing homes etc.

      Other (specify) _Students may recruit through friends, roommates, and classmates

    2. Describe how and by whom the potential participants will be recruited.

      Provide a copy of any materials to be used for recruitment (e.g. posters(s), flyers, advertisement(s), letter(s), telephone and other verbal scripts).

      Since the majority of the recruitment will be done directly by the SD 348 students, a short "email" message has been prepared that they can send to personal contacts. See Appendix G.

    3. Where will the study take place?
      [ X ] On campus Location __EL 108 (computer lab)
      [ ] Off campus Location __________________________

  10. Compensation of Participants
    Will participants receive compensation (financial or otherwise) for participation? Yes [ ] No [ X ]

    If Yes, provide details:

  11. Feedback to Participants
    Briefly describe the plans for provision of feedback. Where feasible, a letter of appreciation should be provided to participants. This also should include details about the purpose and predictions of the study, and if possible, an executive summary of the study outcomes. Provide a copy of the feedback letter to be used.

    See Appendix F

  12. POTENTIAL BENEFITS FROM THE STUDY

    1. Identify and describe any known or anticipated direct benefits to the participants from their involvement in the project.

      The participants will experience usability testing and their feedback and efforts will contribute to the evaluation and redesign of the UW Library Gateway Page.

    2. Identify and describe any known or anticipated benefits to society from this study.

      Ultimately usability improvements to the UW Library Gateway Page will allow the larger community greater ease in accessing UW Library information and resources.

  13. POTENTIAL RISKS FROM THE STUDY

    1. For each procedure used in this study, provide a description of any known or anticipated risks/stressors to the participants. Consider physiological, psychological, emotional, social, economic, legal, etc. risks/stressors. A study-specific medical screening form must be included when physiological assessments are used and associated risk(s) to participants are minimal or greater.

      [ X ] No known or anticipated risks

      Explain why no risks are anticipated:

      Participants will be asked to carry out brief tasks using a webpage. All participants will be familiar with using webpages. Participants will be reminded that their performance is being observed to test the effectiveness of the designs - not to test the skills of the users.

      [ ] Minimal risk

      Description of risks:

      [ ] Greater than minimal risk

      Description of risks:

    2. Describe the procedures or safeguards in place to protect the physical and psychological health of the participants in light of the risks/stresses identified in D1.

      All test monitors will follow the test monitor script to ensure that all participants are instructed in the same manner.

  14. INFORMED CONSENT PROCESS

    1. What process will be used to inform the potential participants about the study details and to obtain their consent for participation?

      [ X ] Information letter with written consent form; provide a copy

      [ ] Information letter with verbal consent; provide a copy

      [ ] Information/cover letter; provide a copy

      Other (specify) ________________________________________________________________

    2. If written consent cannot be obtained from the potential participants, provide a justification.

       

  15. ANONYMITY OF PARTICIPANTS AND CONFIDENTIALITY OF DATA

    1. Explain the procedures to be used to ensure anonymity of participants and confidentiality of data both during the research and in the release of the findings.

      All participants will be assigned a participant number which will be used on data collection forms.

      Data will be aggregated such that no participant will be identified in any of the reports.

    2. Describe the procedures for securing written records, questionnaires, video/audio tapes and electronic data, etc.

      Teams will be responsible for keeping data collection forms until the data collection and analysis is complete.

      Information/consent forms with participant names will be kept in a file separate from the data collection forms to protect anonymity. Once data collection is finished and aggregated raw data (i.e. consent forms, data collection sheets and questionnaires) will be turned over to Prof MacGregor.

    3. Indicate how long the data will be securely stored and the method to be used for final disposition of the data.

      [ ] Paper Records

      [ ] Confidential shredding after ______ years

      [ ] Data will be retained indefinitely in a secure location

      [ X ] Data will be retained until completion of specific course.

      [ ] Audio/Video Recordings

      [ ] Erasing of audio/video tapes after ______ years

      [ ] Data will be retained indefinitely in a secure location

      [ ] Data will be retained until completion of specific course.

      [ ] Electronic Data

      [ ] Erasing of electronic data after ______ years

      [ ] Data will be retained indefinitely in a secure location

      [ X ] Data will be retained until completion of specific course.

      [ ] Other (Provide details on type, retention period and final disposition, if applicable)

       

      Researchers must ensure that all supporting materials/documentation for their applications are submitted with the signed, hard copies of the ORE form 101/101A. Note that materials shown below in bold are required as part of the ORE application package. The inclusion of other materials depends on the specific type of projects.

      Please check below all appendices that are attached as part of your application package:

      [ X ] Recruitment Materials: A copy of any poster(s), flyer(s), advertisement(s), letter(s), telephone or other verbal script(s) used to recruit/gain access to participants.

      [ X ] Information Letter and Consent Form(s)*. Used in studies involving interaction with participants (e.g. interviews, testing, etc.)

      [ ] Information/Cover Letter(s)*. Used in studies involving surveys or questionnaires.

      [ ] Parent Information Letter and Permission Form*. For studies involving minors.

      [ ] Medical Screening Form: Must be included for all physiological measurements and tailored for each study.

      [ X ] Data Collection Materials: A copy of all survey(s), questionnaire(s), interview questions, interview themes/sample questions for open-ended interviews, focus group questions, or any standardized tests.

      [ X ] Feedback letter *

      [ ] ORE Form 102: To be submitted by applicants who wish access to students and/or teachers from the local school boards.

      [ ] Other: _____________________________________________________________________________

       

INVESTIGATORS’ AGREEMENT

I have read the Office of Research Ethics Guidelines for Research with Human Participants and agree to comply with the conditions outlined in the Guidelines. In the case of student research, as a Course Instructor, my signature indicates that I have read and approved the application and proposal, deem the project to be valid and worthwhile, and agree to provide the necessary supervision of the student(s).


_____________________________ _March 15, 2001________

Signature of Course Instructor Date

____________________________________ _________________________

Signature of Student Investigator(s) Date


____________________________________ _________________________

Signature of Student Investigator(s) Date


____________________________________ _________________________

Signature of Student Investigator(s) Date

 

 

FOR OFFICE OF RESEARCH ETHICS USE ONLY:

 

 

_____________________________ _________________________

Susan E. Sykes, Ph.D., C. Psych. Date

Director
Office of Research Ethics

SYDE 348 Winter 200

Instructor:

Carolyn MacGregor Systems Design Eng cgmacgre@engmail.uwaterloo.ca

Teaching Assistants:

Scott Anderson Systems Design Eng sjanders@engmail.uwaterloo.ca

Lora Bruyn Systems Design Eng le2bruyn@engmail.uwaterloo.ca

 

Class List

Student Name Department Email

Minh Cao Systems Design Eng mcao@engmail.uwaterloo.ca

Justina Chan Systems Design Eng jpwchan@engmail.uwaterloo.ca

Cecilia Chung Systems Design Eng cchung@engmail.uwaterloo.ca

Ian Dai Systems Design Eng y2dai@engmail.uwaterloo.ca

Tim Filier Kinesiology tfillier@ahsmail.uwaterloo.ca

Ryan Finnie Kinesiology rlfinnie@ahsmail.uwaterloo.ca

Ankur Gupta Kinesiology a8gupta@ahsmail.uwaterloo.ca

Phat Ha Kinesiology phha@ahsmail.uwaterloo.ca

Veronica Haliniak Kinesiology vmhalini@ahsmail.uwaterloo.ca

Danny Ho Systems Design Eng ycdho@engmail.uwaterloo.ca

Nadim Jamal Systems Design Eng njamal@engmail.uwaterloo.ca

Shawn Kavanagh Kinesiology smkavana@ahsmail.uwaterloo.ca

Chris Klachan Kinesiology cdklacha@ahsmail.uwaterloo.ca

Gerald Lai Kinesiology gelai@ahsmail.uwaterloo.ca

Frank Lardi Kinesiology flardi@ahsmail.uwaterloo.ca

Pamela Lauz Systems Design Eng plauz@engmail.uwaterloo.ca

Arthur Law Systems Design Eng alaw@engmail.uwaterloo.ca

Rhoda Lee Kinesiology r6lee@ahsmail.uwaterloo.ca

General Leung Systems Design Eng g2leung@engmail.uwaterloo.ca

Dylan Lum Systems Design Eng dlum@engmail.uwaterloo.ca

Jasdeep Madpuri Kinesiology jkmadpur@ahsmail.uwaterloo.ca

Carrie Ng Kinesiology cgng@ahsmail.uwaterloo.ca

Scott Nisbet Kinesiology ssnisbet@ahsmail.uwaterloo.ca

Barry Piquet Systems Design Eng brpiquet@engmail.uwaterloo.ca

Kelly Jo Shipp Systems Design Eng kjshipp@engmail.uwaterloo.ca

Robert Snow Systems Design Eng rjsnow@engmail.uwaterloo.ca

Greg Stefan Kinesiology gastefan@ahsmail.uwaterloo.ca

Melissa Thomas Kinesiology m2thomas@ahsmail.uwaterloo.ca

Julia Thompson Kinesiology je4thomp@ahsmail.uwaterloo.ca

Anna Tran Kinesiology a4tran@ahsmail.uwaterloo.ca

Justine Yau Kinesiology jyau@ahsmail.uwaterloo.ca

Simon Yu Kinesiology s2yu@ahsmail.uwaterloo.ca

 

APPENDIX A - Information/Consent Form

SD 348 Winter=01 Team Project

Information Letter

 

RESEARCHERS: Student Members of SD 348 (User-Centred Design), in co-operation with

Prof. Carolyn MacGregor (Systems Design Engineering) and

the members of UW Library Community Needs Assessment Group (CNAG)

TITLE: Usability testing of UW Library Gateway Web Page

STUDY OBJECTIVES:

The class members of SD 348, along with course instructor Prof. Carolyn MacGregor, are working with CNAG to help improve the design and layout of the University of Waterloo's Library Gateway (main) web-page.

The main objective of this usability testing is to evaluate the logic and layout of the current UW Library Gateway web page and to compare it with some proposed redesigns.

TASKS:

All participants will complete a Background Questionnaire. The purpose of this questionnaire is to collect general information on the demographics and computer & library experience of the participants involved in this phase of the usability testing.

Following the questionnaire, you will be asked to carry out a set of specific tasks. At the end of the tasks, we will ask you some general questions on your opinions of how user-friendly you found the system.. You may leave unanswered any questions you prefer not to answer.

Keep in mind that the testing is not an evaluation of how well you might use the system, but whether the web-pages are user-friendly at this phase in the design development.

TIME COMMITMENT:

This study will require approximately 20-30 minutes of your time. Your participation is strictly voluntary.

RISKS & BENEFITS:

There are no known risks to yourself. The benefits of participating in this study is that you will be contributing to the development of a more user-friendly design for the UW Library Gateway web page

RIGHT TO WITHDRAWAL:

You may withdraw from this study at any time by advising the researcher of your decision. At the time of withdrawal, all data collected during your session will be destroyed.

CONFIDENTIALITY:

All data collected will be kept confidential, and you will not be personally identified in any reports. A participant code will be used on all data sheets instead of your name. The only people who will have access to the original data will be the researchers. All data will be destroyed at the end of the course requirements (April 2001).

ETHICS REVIEW:

If you have any questions about participation in this study, please feel free to ask the researchers. If you have additional questions at a later date, please contact Professor MacGregor at 888-4567 ext 2897.

This project has been reviewed by, and received ethics clearance through the Office of Research Ethics at the University of Waterloo. If you have any questions or concerns resulting from your participation in this study, please contact Dr. Susan Sykes at 885-4567 ext. 6005.

 

Consent of Participant

I have read and understood the information presented in the participant information letter and understand the procedures and risks involved in the study. I have received satisfactory answers to my questions related to this study. I understand that this project has been reviewed by, and received ethics clearance through, the Office of Research Ethics at the University of Waterloo. I further understand that if I have any questions or concerns resulting from my participation in this study, I may contact the Director, Office of Research Ethics, at (519) 888-4567 ext. 6005. I am aware that I may withdraw from the study at any time without penalty. With full knowledge of all foregoing, I agree, of my own free will, to participate in this study.

 

_____________________________

Print Name Signature of Participant Dated at Waterloo, Ontario Witnessed

Witness: __________________________ Signature of Witness: __________________

 

Do you wish to receive feedback on the results of this study? ___ No ___ Yes

If yes, please include an email or regular mailing address where a summary of the results can be sent?

Email: ________________________

APPENDIX B - Background Questionnaire

USABILITY TESTING OF UW LIBRARY GATEWAY WEB PAGE.

Test Monitor: _____________ Participant # _____________

 

BACKGROUND QUESTIONNAIRE

General Questions:

Gender: ___ Male ___ Female Age: ____

Program: ___________________ Year: ____ Co-op: ___ Yes ___ No

On a scale of 1 to 5, how familiar would you say you are with everyday computing activities (e.g. word processing, email)?

1
2
3
4
5
not at all
familiar		      
very familiar

On a scale of 1 to 5, how familiar would you say you are with the use of web-pages for on-line activities (e.g. searching for information)?

1
2
3
4
5
not at all
familiar		      
very familiar

On a scale of 1 to 5, how familiar would you say you are with the design and creation of web-pages?

1
2
3
4
5
not at all
familiar		      
very familiar

On a scale of 1 to 5, how familiar would you say you are with UW's Library web-pages?

1
2
3
4
5
not at all
familiar		      
very familiar

On a scale of 1 to 5, how familiar would you say you are with UW's Library services?

1
2
3
4
5
not at all
familiar		      
very familiar

 

APPENDIX C - Test Monitor Scripts

SD 348 Winter=01 Team Project

TEST MONITOR SCRIPTS

 

Note to Test Monitors:

Please read the text in Times-Roman (this font) out loud to each participant. The text in capitals (Arial) are instructions to you and should not be read out loud.

LIST OF USER-TESTING TASKS:

Instructions to Participant, Information/Consent & Background Questionnaire (5 minutes)

Library Tasks using current design or redeisgn (10-20 minutes)

Usability Questionnaire (5 minutes)

 

EQUIPMENT YOU NEED TO BRING TO TESTING SESSIONS:

  • testing script
  • data collection forms
  • a watch to time events
  • clipboard (or something to write on)
  • extra paper for making notes

Step 1: INSTRUCTIONS TO PARTICIPANTS.

NOTE: MAKE SURE THAT WINDOW FOR WEB PAGE IS MINIMIZE SO THAT PARTICIPANT CANNOT SEE IT UNTIL THE TRIALS ARE READY TO BEGIN

Thank you for agreeing to participate in our study entitled AUsability Testing of the UW Library Gateway Web Page@.

This study is being carried out by the class members of SD 348 (User-Centred Design) as part of our course requirements. In order to make sure that all participants receive the same information, I am going to read to you from this script.

We will start by going over the information letter for this study. It will explain the objectives of this study and the tasks that we will ask you to perform.

GO OVER INFORMATION LETTER, AND ASK IF PARTICIPANT HAS ANY QUESTIONS?

ASK PARTICIPANT TO READ OVER AND SIGN CONSENT FORM.

Step 2: BACKGROUND QUESTIONNAIRE

Before we have you use the UW Library Gateway Web Page we would like to ask you some general questions about yourself and your computing experience.

ASK STUDENT TO FILL IN THE BACKGROUND QUESTIONNAIRE.

Now for the main part of the study.

Step 3: LIBRARY TASKS

Before we start, I need to remind you that we are testing the usability of the web page design and not your ability to use the system. I will be taking notes as you work through the tasks and recording how long it takes someone to find the appropriate links and where people may have difficulties using the web page. This will help us to improve upon our design before our presentations next week.

I will be asking you to work through a series of tasks that people normally associate with libraries. We will be going through each task one right after the other and then I will give you an opportunity to comment about the web page and tasks once they are all done.

For each task, you will start with your hand on the mouse and the cursor at the bottom centre of the screen. I will then read the task out loud and once I am finished you are to try to find the appropriate link on the web page as quickly as possible. I will tell you when you have found the appropriate link which will end that task. I will then remind you to move the cursor to the bottom centre of the screen before I read the next task.

  • BRING UP WINDOW AND ASK PARTICIPANT TO PLACE HAND ON MOUSE AND MOVE CURSOR TO BOTTOM CENTRE OF SCREEN.
  • READ FIRST TASK FROM SHEET.
  • START TIMER.
  • STOP TIMER WHEN PARTICIPANT HAS CLICKED ON APPROPRIATE LINK THAT MOVES USER OFF OF THE GATEWAY.
  • NOTE ANY PROBLEMS ON OBSERVATION SHEET BEFORE MOVING TO NEXT TASK.
  • REMIND PARTICIPANT TO MOVE CURSOR TO BOTTOM CENTRE AND KEEP HAND ON MOUSE WHILE YOU READ NEXT TASK OBJECTIVE.

ONCE ALL TASKS ARE COMPLETED, ASK PARTICIPANT TO FILL OUT USABILITY QUESTIONNAIRE.

Step 4: USABILITY QUESTIONNAIRE

As a final task we would like you to answer some questions about the usability of the UW Library Gateway Web-Page design that you just used.

 

APPENDIX D - Data Colletion Sheet

UW LIBRARY GATEWAY WEB PAGE

DATA COLLECTION SHEET

Test Monitor: ________________ Participant #: ________________

RECORD TIME AND OBSERVATIONS

TASK

TIME (seconds)

COMMENTS

1

 

 

 

2

 

 

 

3

 

 

 

4

 

 

 

5

 

 

 

6

 

 

 

7

 

 

 

8

 

 

 

9

 

 

 

10

 

 

 

APPENDIX D (con’t) Data Collection Task List

VERSION A

Task

Objective: Find the links that would allow you to answer the following ...

   

1

Where would you go to find general help with a research topic in your department?

2

How do you get an article from a journal that is available at WLU?

3

Can alumni borrow books from UW?

4

What are the hours for the University Archives?

5

What books are available at the Kitchener Public Library?

6

Does the library have any electronic dictionaries?

7

Does UW library have a copy of Andrew Pyper’s Lost Girls?

8

Where can you find instructions for connecting from home?

9

Where would you go to renew books online?

10

What is the contact information for the Liaison Librarian assigned to your home department?

   
VERSION B

Task

Objective: Find the links that would allow you to answer the following ...

   

1

What is the contact information for the Liaison Librarian assigned to your home department?

2

Where would you go to renew books online?

3

Where can you find instructions for connecting from home?

4

Does UW library have a copy of Andrew Pyper’s Lost Girls?

5

Does the library have any electronic dictionaries?

6

What books are available at the Kitchener Public Library?

7

What are the hours for the University Archives?

8

Can alumni borrow books from UW?

9

How do you get an article from a journal that is available at WLU?

10

Where would you go to find general help with a research topic in your department?


 

APPENDIX E - Usability Questionnaire

UW LIBRARY GATEWAY WEB PAGE

USABILITY QUESTIONNAIRE

Test Monitor: __________________ Participant # _______________

 
  1. On a scale of 1 to 5, how USER-FRIENDLY would you say the web-pages were in terms of conveying which actions you needed to perform to complete the following tasks?

     
    Not at all
    User-friendly
    		  

    Very user-
    Friendly
    Key words
    9
    9
    9
    9
    9
    Major Links
    9
    9
    9
    9
    9
    Graphics
    9
    9
    9
    9
    9
    Navigation Bars
    9
    9
    9
    9
    9
    Overall Layout
    9
    9
    9
    9
    9

  2. A) In terms of overall usability, how would you compare this web page with the current UW Library Gateway Page

    9 not applicable, (haven't used current UW Library Gateway Page) --go to #3
    9 much better than current UW Library Gateway Page -- go to #3
    9 marginally better than current UW Library Gateway Page -- go to #3
    9 about the same as current UW Library Gateway Page -- go to #3
    9 marginally worse than current UW Library Gateway Page (go to # 2b)
    9 much worse than current UW Library Gateway Page (go to #2b)

    B) In your opinion, what would you say are the major problems with the new design that makes it worse than the current current UW Library Gateway Page?

  3. Do you have any additional comments or feedback on the web-pages you used today?

 

APPENDIX F - Feedback Letter

USABILITY OF THE UW LIBRARY GATEWAY WEB PAGE

March 22, 2001

Dear Participant:

Thank you for helping us with our usability study as part of our course requirements for SD 348 (User-Centred Design). The class has been working with the UW Library's Community Needs Assessment Group to help improve the usability of UW's Library Gateway Web Page. UW would like to make sure that its library resources and services are accessible and easy to find for UW students and faculty, UW alumni, and the larger community. Improving the usability of the library web pages will help them to meet that goal.

There are six different teams in our class. Each team has been working on an alternative design for the current UW Library Gateway Web Page. We will be using the information that we have gathered from you along with the other participants to help improve upon the usability of our web page designs. As stated in the information letter, the data from participants will be analysed as a group and you will not be identified personally.

As a class we have been following a user-centred approach to design. Each team has developed their own prototype of the UW Library Gateway that has evolved over the course of the term through usability analysis and testing. Our class will be giving a presentation of the evolution of our web designs on

Tuesday, March 27, 2001, at 9 am - 10:30 am in E2 1303 B.

You are welcome to attend to see how your input may have helped with the development of the final designs.

If you would like further information about this project, please contact Prof. Carolyn MacGregor at 888-4567 ext 2897 or at cgmacgre@engmail.uwaterloo.ca

If you have concerns regarding your participation in this study, please contact Dr. Susan Sykes at the Office of Research Ethics, 888-4567 ext 6005.

Thank you again for your participation.

The Students of SD 348.

 

Appendix G. - Email "flyer"

WANTED: VOLUNTEERS FOR

Usability Testing of UW Library Main Gateway Web Page

ARE YOU A UNIVERSITY OF WATERLOO STUDENT?

DO YOU HAVE 20-30 MINUTES

TO TEST THE USABILITY OF THE UW LIBRARY GATEWAY WEB PAGE???

The class members of SD 348, along with course instructor Prof. Carolyn MacGregor, are working with the UW Library Community Needs Assessment Group to help improve the design and layout of the UW Library Main Gateway Web Page.

UW Library has lots of great resources on-line and wants to make sure that students can easily find that information.

This is your chance as a UW student to help with the preliminary user testing of the current and new web-page designs.

If you are interested in participating, please contact [student puts email address here] or if you have further questions please contact Prof. MacGregor at cgmacgre@engmail.uwaterloo.ca

This project has been reviewed by, and received ethics clearance through, the Office of Research Ethics at the University of Waterloo. If you have any questions or concerns resulting from your participation in this study, please contact the Director, Office of Research Ethics, at (519) 888-4567 ext. 6005.


Close this window