INTERIM REPORT NO. 3 – Final Report
UNIVERSITY OF WATERLOO
LIBRARY INTERNET GATEWAY
SYDE 348
Prof. Carolyn MacGregor
Project Manager: Lora Bruyn
Client Contact: Anne Fullerton
Submitted By:
Ankur Gupta |
98.034.751 |
Veronica Haliniak |
98.184.986 |
Pamela Lauz |
94.040.936 |
Barry Piquette |
95.104.651 |
Julia Thompson |
97.192.541 |
Simon Yu |
98.431.554 |
Tuesday, March 27, 2001
Table of Contents
- Executive Summary
- INTRODUCTION to the Problem
- Interactive Systems Problem Statement (ISPS)
3.1 Breakdown of ISPS 3.1.1 Human Activity
3.1.2 Users
3.1.3 Level of Support
3.1.4 Form of Solution - Project Constraints and Requirements
- Phase 1 UCD Methods
5.1 Intent of Methods 5.1.1 Design Walkthroughs 5.2 Results and their Design Implications
5.1.2 Heuristic Evaluation
5.1.3 Hierarchical Task Analyses
5.1.4 Competitive Analyses - Phase 2 UCD methods
6.1 Intent of Methods 6.1.1 Card Sorting 6.2 Results and their Design Implications
6.1.2 Discount UsabilityHeuristic Evaluations
Rationale
Method
Design Walkthroughs
Rationale
Method
- Phase 3 ucd methods
7.1 Keystroke-Level Analysis 7.1.1 Rationale 7.2 Lab-Based Usability Testing
7.1.2 Procedure
7.1.3 Keystroke Level Analysis (KLA) Results7.2.1 Rationale 7.3 Design Implications According to Results from Lab-based Testing
7.2.2 Procedure
7.2.3 Results
7.2.4 Usability of Gateway
7.2.5 Additional Comments
7.2.6 The Proposed Gateway vs. the Current UW Library Gateway - Final Design Specifications
- References
- APPENDICES Table of Figures and Tables
Figure 1: Initial stage of the Prototype 1 gateway.
Figure 2: Stage two of the Prototype 1 gateway.
Figure 3: Stage three of the Prototype 1 gateway.
Figure 4: Medium Fidelity Prototype
Table 1: Significant differences between the actual and theoretical (keystroke analyses) values for each question asked during the lab-based testing.
Table 2: Relation ship between User-friendly, Neutral, and Non-user Friendly for different usability characteristics on the proposed gateway.
Table 3: Rating of proposed gateway relative to current UW gateway
- Executive Summary The current University of Waterloo library gateway fails to capitalise on user centred design principles. Minimal usability testing was carried out in the design of this site and therefore more extensive testing was necessary in order to provide a more user-friendly interface. The Community Network Action Group (CNAG) approached the design team, requesting design recommendations to make the gateway a more useful and intuitive for users.
- INTRODUCTION to the Problem Over the past few years, the use of the Internet has provided business, community, and educational organizations with an effective medium for accessing large amounts of information and services. For example, business organizations such as Air Canada and the Canadian Imperial Bank of Commerce (CIBC), structure their homepages to provide access for customers to commonly used services in an attempt to expedite business transactions. On the other hand, community organizations such as the City of Toronto, exploit the Internet by providing a medium for community members to access information regarding community services, events, and contact information. Alternatively, educational organizations such as the University of Waterloo, have capitalized on this information-transferring tool by providing students and faculties with access to resources such admission information, faculty web pages, course web pages, housing and residence services, and the university library (http://www.lib.uwaterloo.ca). In fact, the library is a gateway to access both academic and non-academic information. As such, it is of utmost importance that the University of Waterloo library (UWL) gateway page is maintained as a web site that can provide valuable information in a user-friendly manner.
- Interactive Systems Problem Statement (ISPS) The overall purpose of this project was to redesign the UWL gateway so that it is useful, effective, and easy to learn for retrieving and accessing academic and non-academic information and to support all levels of users from novice to experts including undergraduate and graduate students, faculty members, staff administrators, system developers, and local community members.
- Project Constraints and Requirements The overall focus of the design project was to address concerns related to the design of the UW library gateway in order to make it useful, effective, and easy to learn for users to retrieve and access academic and non-academic information. Design considerations were limited to those aspects outlined by the client at the beginning of the assignment and included the overall layout of the gateway, graphics, mouse-overs vs. text layout, dead spaces, language, navigation, layout consistency, and clarity of the top navigation bar.
- Phase 1 UCD Methods This section of the report is only a summary of the information contained in the first interim report. For more detail and explanation regarding the methods and design implications, please refer to Interim Report 1.
- Find an e-journal on Ergonomics
- Find out if there are any overdue books on your account
- Find out if there are any course reserves for any of your courses
- Can you access an e-journal article from home?
- Does the library have any electronic dictionaries?
- Visibility of System Status: The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
- Match between System and the Real World: The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
- User Control and Freedom: Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
- Consistency and Standards: Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
- Error Prevention: Even better than good error messages is a careful design which prevents a problem from occurring in the first place.
- Recognition Rather than Recall: Make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
- Flexibility and Efficiency of Use: Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
- Aesthetic and Minimalist Design: Dialogues should not contain information that is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
- Help Users Recognize, Diagnose, and Recover from Errors: Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
- Help and Documentation: Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.
- Phase 2 UCD methods Similar to the previous section, Phase 2 UCD Methods is a summary of Interim Report 2. For more detail and description than is covered here please refer to Interim Report 2.
- Phase 3 ucd methods
- The test monitor reads the question to the user, and that the trial starts once the question has been read.
- The user has his/her hand on the mouse as the question is being read (i.e. don’t need to calculate in initial "home" to device).
- The user keeps his/her hand on the mouse throughout the trial.
- The cursor is at the centre bottom of the screen at the start of the trial.
- The user is an average typist (i.e. Average skilled typist = 0.20 sec per keystroke).
- The user is experienced (i.e. does not make errors).
- The response by the system was negligible.
- Have a "services" section for connecting from home etc.
- Library and Community info seems very broad and maybe better separated into Library Info, which is for more important; and Community Info (user was unsure why people would visit the library site for community information).
- Library Info is pretty major, yet it is buried in the middle: it should go before "Other Libraries"
- Links can be a little more distinct so there’s less time to search for things
- "How Do I?" is brilliant, yet it may be better to locate it on the side above the "Search the Library Site" instead of at the top since the navigation bar gets lost in the banner.
- Final Design Specifications The proposed UW Library Gateway Web Page can be seen in Figure 5. All aspects of the gateway are a result of the 3 phases of UCD methods and testing as previously summarized and as outlined in detail in interim reports one and two. It must be noted that as the project progressed through the many iterations, not only were recommendations made to add new items, but also to remove some recommended features that were later considered non-userfriendly. The final product is the result of all the additions and removals of existing components of the site and any recommended through the iterations of the prototype. The final specifications of the site will be discussed in detail to serve as a summary of all the changes that are seen in the proposed UW Library Gateway Web Page.
- References Chi, Chia-Fen and Chung, Ku-Lun (1996). Task analysis for computer-aided design (CAD) at a keystroke level. Applied Ergonomics, 27 (4), pp. 255-265.
- APPENDICES
The team developed design requirements based on CNAG’s gateway constraints and criteria. This report contains the methodology and results obtained through usability testing that are applied to final prototype. In phase 1 of the design process the team employed design walkthroughs, heuristic evaluations, hierarchical task analyses, and competitive analyses to discover problem areas associated with the current site. Semantics, navigation, and overall layout were addressed during this phase. Phase 2 of the design process involved testing users to discover additional usability issues. Through Card Sorting, available library link headings were grouped to improve user- intuitive searching. Discount usability techniques eliminated an alternative design proposal as well as further improving heuristic measures and layout. The third phase involved applying keystroke analyses and lab-based usability testing on the medium fidelity prototype. Aesthetic details and semantics were altered using the results of these methods.
This report includes the three progressive prototype iterations and the reasons for improvement after each design cycle of the spiral design model. The proposed UW Library Gateway Web Page is the product of all the UCD methods applied to the prototype iterations. The final design specifications of the gateway include enhanced feedback mechanisms, better use of semantics, increased feature salience, and more intuitive content layout.
Presently the library spends an approximate 5.7 million dollars annually to provide a variety of resources for library users (Anne Fullerton, February 1, 2001), which include access to academic and electronic journals. However, the University of Waterloo Community Network Access Group (CNAG) is concerned that the current UWL web site interface is not user-friendly and valuable information is going unnoticed or unused by many of its users. As a result, CNAG was interested in developing a web-site that would be more useful and effective for students, faculty, and other web browsers to retrieve academic and non-academic information. Their main concerns regarding the re-design of the current site included: the overall site-layout of the library gateway, graphics on the gateway, mouse-over versus a text layout, dead or white space, semantics and jargon of labels, navigation, layout consistency, and clarity of the tool bar(s).
3.1 Breakdown of ISPS
The UWL gateway will be mainly employed by users to locate books, journals, course reserves, and other relevant research information at the University of Waterloo or surrounding TriUniversity Group of Libraries such as Wilfred Laurier University or the University of Guelph. The information that is displayed should provide sufficient information to users regarding the specific location or call number of where these resources can be found. In addition, users should be able to place holds on these resources or retrieve these resources from any site of the TriUniversity Group of Libraries.
Assumptions
Users will be assumed to be operating a computer on or off campus to perform a variety of library tasks including locating information about library ervices, checking personal account information, or finding out library hours. It will also be assumed that although most users will use the gateway page(s) for a short period of time, some users may spend a great deal of time interacting with the web site interface especially if they are unsure of what specific information they are looking for and where it can be found. Furthermore, it will be assumed that users may not be searching the library site to obtain library materials but for other non-library resources such as web searches or using electronic dictionaries.
Users of the UWL gateway will mainly be University of Waterloo students (undergraduates, graduates, and distance education students), staff, and faculty members. The level of experience will range between first-time users who have never used the site before and expert users who frequently visit the UWL gateway to access relevant information.
Assumptions
In addition to local community members, it will be assumed that students, staff, and faculty from other educational institutions outside the University of Waterloo will be accessing services and resources from the UWL gateway. Users will be assumed to have a general knowledge of library services (e.g. search for books, journals, etc.) with at least a novice level of computer experience in how to use a mouse, the keyboard, and the Internet. In terms of language, users will be assumed to be proficient in reading, writing, speaking, and understanding English at no less than a grade 6 level. The users will be assumed to be average typists with sufficient physical ability to use computers (e.g. sight, physical dexterity, etc.).
Users of the UWL gateway should find the web site quick and easy to use or navigate. The system interface should be easily understood with help being available for users having trouble locating or utilizing library services and resources.
Assumptions
It will be assumed that users do not want to devote a large amount of time learning how to use or interact with the web site. In other words, it will be assumed that most users will want to "walk-up and use" any computer terminal on or off campus to accomplish their tasks with very little difficulty (i.e. the system’s interface should be "user-friendly", easy to use and learn, non-intimidating, concise, and intuitive.). The help menus and topics should be easy to use and understand and allow users to be able to solve their problems on their own, especially those users accessing library services and resources from remote locations where immediate assistance may not be available. Furthermore, it will be assumed that effective error recovery should be available should users experience any such problems.
Assumption
It will be assumed that the majority of users will have the functional computer system requirements and preference for a graphical web page. However, it will be assumed that some computers may have incompatible operating systems to display these web pages, and that access to a textual version of the web page should be provided. Furthermore, it will be assumed that an aesthetically pleasing interface will be important to users of the system, especially in terms of intuitive and legible text (i.e. font selection and font size) and easy to read page layouts (i.e. spacing, overlap, etc.).
Additional concerns related to the use of the UWL gateway were identified during client meetings, several usability inspection methods, and a competitive analysis of other library web sites. These concerns were amalgamated and categorized according to accessibility, semantics, navigation, functionality, aesthetics, and online help tutorials. As such, this information was the basis for the design requirements to develop the various fidelity prototypes.
The issue of accessibility was a concern for providing an environment that affords both novice and expert users a convenient method to easily and efficiently use resources and acquire necessary information. In terms of the UW library gateway page, there are several implications of accessibility with the use of both the text-version and graphical version. Although less overwhelming, the graphical version is criticized for making it more difficult to locate information quickly and efficiently. The headings used do not present the user with a clear understanding of what information can be found within each category. The categories were reorganized, stating headings clearly and unambiguously. This restructuring helped to improve search time for users.
The issue of semantics regarded the implicit and explicit description of headings, icons, and graphics used throughout the web site. Specifically, the wording used in the UW library gateway was not effective in conveying the essence of each heading and was in many regards, confusing. As a result, headings were renamed and affiliated content regrouped.
The issue of navigation was twofold. One concern involved issues of accessibility and providing users with a relatively easy system to navigate through the site. This would allow the users to efficiently locate necessary resources and/or information. The second issue was concerned with providing users with a method of returning to main pages as the user got deeper and deeper into the hierarchical levels of the web page.
A specific problem related to the functionality of the current library gateway involved concerns with the graphical version. It was made apparent that users have difficulty understanding the mouse-over procedure, especially with the inconsistency in which these procedures were used. It was suggested that better mappings between the category headings and the pop-up boxes be made more prominent. Better affordances with the mouse-over technique were also emphasized. Concerns with aesthetics were concentrated on the effects of contrast, background images, font size, and colours and how these factors affected the visual display of information and functionality of the interface to the user. Additional considerations were given to the colour on the buttons and to the font colours.
Although not a specific concern, a common theme between gateway pages was the use of some form of a "Frequently Asked Questions" link. This link could take the form of "Tip of the Day", "Ask the Librarian", or "How do I…". This concern seemed extremely beneficial to novice users who may be unaware of certain features of the library gateway or who are unable to accomplish their goals with the site. In addition, an alternative to a specific "help section", such as "tip of the day", was thought to be less intimidating and therefore would be more frequently used.
Due to time constraints, this project did not undertake the redesigning of the text-based version of the gateway. However, the text-base version was reviewed however to view the effectiveness of a highly text oriented web page. As a result, many of the semantic changes made to links on the graphical gateway could also be implemented to the text-based gateway. Time constraints also prevented the designers from carrying out user testing on large samples of users from a wide variety of populations. There were also financial constraints that restricted much of the user testing to be carried out through pen and paper sketches or other simplified methods.
5.1 Intent of Methods
Rationale
A design walkthrough was conducted on the existing library gateway to gain a better understanding of current interactivity practices as well as users’ subjective feelings regarding the existing UW library gateway. The information obtained from these sessions were expected to provide insight into which areas of the gateway needed to be considered as well as the level of urgency in which these areas should be addressed. Moreover, those areas that needed to be addressed were considered to be those tasks that caused the user to perform an incorrect action to achieve their goal or areas that forced the user to question the information presented.
Method
One member of the design team tested three participants from the Faculty of Engineering with design walkthroughs. While in a campus computer lab, the experimenter asked the participants to perform five tasks on the current UW library gateway:
Heuristic evaluations were implemented in evaluating the UW library gateway because it was a quick method to execute and it did not require specialized knowledge on the part of the evaluators. As well, heuristic evaluations tend to provide information similar to other usability methods such as a checklist approach or SHERPA (Systematic Human Error Reduction and Prediction Approach) (Stanton, 1998). Thus, one heuristic evaluations can be used to collect different types of information which is usually obtained through separate analyses (e.g. system functionality problems and areas for human error).
Method
Nielsen’s ten heuristics (Nielsen, 2001) were applied to the current UW library gateway to analyze its functionality, effectiveness, and usefulness in accomplishing different tasks. Each design team member first used the heuristics to individually evaluate the current UW library gateway before meeting to discuss the usability issues they discovered. Separating the evaluation among team members prior to the group discussion alos functioned to include everybody’s ideas and input without the effects of group polarization, social loafing, or individual dominance. During the group discussion, the major design issues were considered and ideas for their rectification were brainstormed.
Neilson’s heuristic guidelines (Neilson, 2001) are:
Hierarchical tasks analyses (HTAs) use relatively straight forward concepts and serve as sources of input for other predictive methods such as keystroke analyses. Besides offering descriptions of the tasks under investigation, HTAs also help designers to determine specific usability issues that may arise during certain interactions (e.g. tasks, operations) between the user and the system. Some of these usability issues will include safety factors and erroneous actions that need to be addressed in order to maintain safe usage of the product. As a result of these reasons, HTA was a desirable method to analyze the current UW library gateway.
Methods
Many task analyses could be performed for the UW library gateway, however due to time restrictions, only three tasks were investigated. These tasks were chosen from the list of five questions asked during the design walkthroughs and included both tasks that were executed on a frequent basis (e.g. finding an e-journal, renewing books) as well as resources that were infrequently accessed (e.g. electronic dictionaries).
Using the gateway, one design team member determined the break down of each task. With the help of another design team member, solutions for problematic situations were developed. For a description of each task and their respective HTA, please refer to Interim Report
Rationale
In order to recommend changes for the current UW library gateway design, it was important to evaluate the work already done on the current design as well as the work of other library gateways of several different North American Universities. Analysis of the competitor sites provided an understanding of their strengths and weaknesses with a main focus of learning what works well and what doesn’t work well. Thus, the goal of the competitive analysis was to compile a set of design requirements that could be considered or should not be considered in implementing the redesign of the UW library gateway. Solely analyzing the strengths and weaknesses of the UWL gateway on its own would have limited these ideas promoting a bottleneck approach towards the implementation of good and bad design features.
Methods
The library gateways under investigation included University of Arizona Library, University of Washington Library, University of Pennsylvania Library, NCSU Library, and Dalhousie University Library. Each design team member visited each library site mentioned and spent a large amount of time analyzing and compiling a list of their respective strengths and weaknesses. These analyses were amalgamated into one large list and upon discussion, the most important strengths and weaknesses were then chosen for further analysis and potential implementation to the redesign of the UW library gateway.
5.2 Results and their Design Implications
Accessibility issues discovered in the UW gateway page limited users in efficiently employing resources and acquiring information. A clear example of this involves viewing the text version of the UW library gateway site. The abundance of information was overwhelming and imposed high cognitive demands on the user, thus decreasing the speed and efficiency of accessing resources from the site. Furthermore, although the graphic interface is much less overwhelming, it prevented the user from easily accessing information due to the ambiguity and lack of clarity in the semantics of main headings. The team took time to reorganize content found within the gateway and added in a search option in attempt to decrease the amount of time spent accessing resources.
The semantics used to label main headings and content links on the gateway were unclear and appeared to confuse users. Although more time was spent on changing the semantics in the second phase of the re-design process, some heading and content names were altered for the first prototype. For example, the "Find it" heading was changed to "Books and Journals" to demonstrate the types of links and resources available within the menu. Changes to the semantics were intended to enhance users’ understanding of the main headings and allow them to access information more quickly and accurately.
Addressing navigational issues involved investigation of the types of tools the site was lacking as opposed to issues already present in the site. For example, there was no means to easily navigate within the site whether it was to search into or backtrack out of the different levels. It became obvious that navigation tools were needed not only to enter the different levels of the site, but also to escape webpages (e.g. in the case of an incorrect entrance to a link). Since these navigational tools were missing from the UW library gateway, buttons to return to the library homepage were intended to be implemented. The "quick search" function was also intended to allow the users to enter the sites without having to follow a long and unnecessary path. Also proposed was a consistent navigation bar for each level within the UW library site. This would allow the user to efficiently access important and frequent information at all times. Furthermore, the use of a hierarchical structure that allowed the user to visualize where they are within the site and how to trace their previous steps would enable quicker navigation and error correction.
In terms of functionality, the graphic interface (rather than the text version) was addressed. There were many problems associated with poor feedback in using the links (within pop-up menus) and the menu headings themselves. Salient feedback upon mousing over of the headings and links was missing from the gateway, as well as the relationship between the pop-up menus and their associated menu headings. This lack of feedback ultimately made the site more difficult to use and prohibited the user from quickly identifying main headings and destinations. To accommodate for these usability issues, the design team implemented a bar connecting the menu heading and the pop-up menu. Furthermore, mousing over the links within the pop-up menus gave no indication of which item would be activated upon clicking the mouse. The design team increased the spacing between the items within the pop-up menus, and added a mechanism to highlight the link upon mouse overs in order to address the confusion. In general, increasing the functionality of the gateway was an attempt to implement of better mappings between headings and pop-up boxes and better button design that would allow for increased affordance.
Although function should precede aesthetics, users will enjoy using a site that is appealing. Issues regarding the aesthetics of the site involved the inherent lack of contrast of the graphic interface. The lack of contrast was attributed to the similar color schemes of the main menu buttons and the watermark thus making the buttons less salient to the user. The lack of salience and contrast does not lend itself to efficient use of the site since the user had to concentrate more on determining what was a link and what was not. The prototype was designed to create contrast between the buttons and the background. This increased clarity offers the users greater efficiency of use and should allow for quicker identification of the headings. Graphics were also eliminated from the initial prototype to keep the iteration simple until the testing and analysis was completed.
A final area of concern was the lack of a visible online help tutorial directly on the UW library gateway. Many other library sites offered such a tool and thus the importance of such a function was more apparent. It was concluded that such a tool would offer a less intimidating means to seek not only help in performing tasks, but also a new way to perform more common ones. In the initial prototype, the help function was added to the navigation bar so that it would be available to the users at anytime during their interactions with the library web-pages.
These above mentioned findings and design modifications were all very critical in the designing the prototype. The initial prototype is depicted in Figures 1, 2, and 3.
Figure 1: Initial stage of the Prototype 1 gateway.
Figure 2: Stage two of the Prototype 1 gateway.
Figure 3: Stage three of the Prototype 1 gateway.
Usability metrics, collected during the first phase of usability testing, were integrated to create an initial low-fidelity prototype. As a result, the second phase of the design cycle was primarily focussed on testing intended users with card sorting and discount usability on various iterations of the initial design specifications. The information collected from these two methods was then used in an iterative approach to evaluate and determine design problems associated with the system’s interface.
6.1 Intent of Methods
Rationale
Card sorting was primarily used to reorganize the semantic content of the UW library gateway to make navigation through the site more intuitive. Of particular interest to the design team was which links were to be placed on the main navigation bar (high priority), which links were to be placed in the main layout (medium priority), and which links were to be placed in the "Weekly Special" area (low priority).
Method
Thirteen UW undergraduate students, alumni, and non-UW students (e.g. Conestoga College) from the faculties of Mathematics, Engineering, Applied Health Sciences, Science, and Arts participated in this study. The "cards" were labeled with links recorded on sticky-notes and randomly positioned on three pieces of lined paper. Additional 8.5" x 11" pieces of paper were divided into 8 groups with a final page divided into three sections (most frequent, moderate frequency, least frequent).
Participants followed a free form method of card sorting where they were instructed to arrange the cards into as many groups as necessary with the stipulation that each group was to contain similar concepts. The users were then requested to prioritize and name these chosen groups. Upon completion, the cards were returned to the content pool in a random order and the subjects were instructed to group cards into most frequently used, mid-frequently used, and seldom or rarely used categories. The findings from both tasks were integrated to reorganize the content of the web site as well as to determine whether or not to include infrequently or mid-frequently used links.
Heuristic Evaluations
As mentioned above, a heuristic evaluation is a usability inspection method that requires "an [expert] analyst to use judgement, intuition, and experience to guide the product evaluation" (Stanton, 2000, p. 24).
Rationale
A heuristic analysis was necessary to provide informative feedback regarding the usability of the two prototypes in a simple, and time- and cost-effective manner. Care should be taken when generalizing these subjective findings since the expert evaluators may have approached the design process from an expert point of view rather than that of a novice user.
MethodThree team members were chosen to perform this usability inspection method for every design iteration. Instead of individually assessing the prototypes with Neilson’s heuristics (2001), the three members worked together to determine conflicts between the prototype and the heuristic guidelines.
The members evaluated pencil and paper sketches (with sticky-note menus) of the prototype interface. Neilson’s (2001) ten usability principles were used to evaluate the prototypes to maintain consistency with the heuristic evaluations completed for Interim Report 1. As well, Neilson’s heuristics are a representative set of standard guidelines recognized in industry. An initial evaluation of each prototype was documented in an attempt to determine which of two prototypes would be used in subsequent design iterations. Two additional heuristic evaluations were conducted on the chosen prototype and changes were made following the completion of each evaluation. Note that each set of heuristic evaluation was followed by an evaluation using design walkthroughs to justify the recommended changes.
Design Walkthroughs
As indicated above, design walkthroughs involve users carrying out tasks on a product. They are often necessary to determine how users will interact with an unfamiliar system interface (Newman and Lamming, 1995), as is the case with our low-fidelity prototype.
Rationale
Design walkthroughs provide an opportunity to investigate the features of a system’s interface through methods of exploratory learning (Newman and Lamming, 1995), particularly by observing users interact with a design prototype. Information provided by this methodology includes users’ initial impressions, task completion time, task completion success rate, frequency of use for online help, and measures of users’ satisfaction.
Method
Two users were randomly requested to participate in the testing of each prototype iteration. In total, six UW undergraduate students (3 males and 3 females) from the faculties of Engineering, Mathematics, Applied Health Sciences, and Arts participated in this study. The same pencil and paper sketches from the heuristic evaluations were used for the design walkthroughs. The testing involved three design walkthroughs where users were instructed to complete the same five tasks that were used in the design walkthroughs for Interim Report 1. Users worked through the tasks by verbalizing their actions and "pushing" the buttons on the prototype. When the user "pushed" a button or "pulled" down a menu, a design team member would place the appropriate menu (sticky-note) on the prototype. Another design team member recorded each action the user performed.
Following completion of the task, the users were requested to provide any additional concerns or problems they encountered while using the prototype, as well as to provide any suggestions for improvement.
Changes were made following each iteration by summarizing the findings from both heuristic evaluations and design walkthroughs. Furthermore, in the initial design walkthrough, users were requested to indicate a preference for one of the two prototypes. As a result, additional design walkthroughs were only conducted on one prototype.
6.2 Results and their Design Implications
More changes to the prototypes were possible from the results obtained by discount usability testing. Through discount usability and lack of user acceptance, it was possible to omit one of two prototypes from further usability testing. The findings are summarized as a result of all the cycles of the method and figure 4 depicts the final medium fidelity prototype.
To make the "How do I…" section more clear to the users as a section that offered help, the addition of a question mark was required. Otherwise, the users admitted their willingness to resort to this function if they so required. However, some of the participants incorrectly used the "Search the Library Site" as they believed it would search, for example, for specific titles of journals or authors of books. To help make the function of the search dialog box more salient, the title was changed to "Site Index", a mechanism to search the various resources within the gateway (e.g. e-journals, library hours, renewals). Semantic problems were also encountered with the "Research Tools" menu heading. This was actually a group of resources developed from the card sort testing, however the menu name was less intuitive than originally believed. The design team motioned to change the heading to read "Online Reference Tools" with the intent to provide users with an better understanding of the contents within the menu. The discount usability testing also revealed the difficulties experienced by the users in locating the link "Course Reserves". For this reason, this link was also added to the content list for the "Resources for People of UW" heading. This change should allow for greater flexibility and easier access to the link. As a final note, a section of the prototype was devoted to a "Weekly Special". This feature would provide links to otherwise seldomly used resources available on the UW library website and increase the usefulness of the site.
Figure 4: Medium Fidelity Prototype
With the completion of two phases of formal usability testing, the final stage provided information regarding the upgrading and maintenance of the existing medium fidelity prototype. Specifically, the effects on learning, throughput, and flexibility were of particular interest to the design team. Preece (1993) defines learnability or ease of learning as the "time and effort required to achieve a specified level of user performance" (p. 47) with throughput referring to the ease of use of a particular system by measuring the speed of task execution and the errors made. Furthermore, Preece (1993) describes flexibility as "the extent to which users can adapt a system to new ways of interaction as they become more experienced" (p.47). As such, in order to incorporate these usability metrics into the design cycle, the final stage of usability testing focused on a keystroke-level of analysis and lab-based usability testing. The findings from both methods were compared in order to outline benchmark designs as well as the functional and design requirements of the interface.
7.1 Keystroke-Level Analysis
7.1.1 Rationale
Members of the design team were responsible for conducting keystroke analyses for ten tasks to be carried out (posed in the form of a question). These tasks were to be used in the usability lab-based testing (please refer to Appendix A). Each design team determined the KLAs for one to two of the tasks under consideration.
Apparatus
The performance times for keystroke-level operators from Card et al (1983), as cited from Newman and Lamming (1995), were used as the basis for this analysis. These tables are reproduced in Appendix B.
Method
Each design team member carried out a keystroke-level analysis by following the procedures for completing the performance times for keystroke-level operators as outlined by the tables from Card et al (1983) cited from Newman and Lamming (1995). The analysis encompassed an assessment for experienced and advanced users using the proposed UW gateway prototype and the site index. The keystroke level analyses (KLA) were only carried out until the keystroke took the user off of the Gateway page In completing this analysis, the following assumptions were made:
7.2 Lab-Based Usability Testing
7.2.1 Rationale
Each design team member was responsible for recruiting 2-3 participants for testing, while additional participants were recruited at the computer lab. In total, 13 participants were studied including 5 males and 8 females. Twelve of the thirteen participants were undergraduate students while the remaining subject was a masters student.
A background questionnaire administered to participants at the beginning of testing revealed that the participants ranged in age between 18 to 37 years, with a mean age of 22.5 years and standard deviation (SD) of 4.93 years. The level of schooling ranged between 1-5 years (mean of 3.08 years and SD of 1.16 years) with subjects being enrolled in the following programs of study: Engineering (Systems Design, Computer Engineering, and Electrical Engineering), Applied Health Sciences (Kinesiology), Arts (Psychology and Economics), Science (Biology, Science-Business), and Mathematics (Computer Science).
A scale ranging from 1 to 5, where 1 represented "not at all familiar" and 5 represented "very familiar", was used to determine the users’ familiarity computer activities and the UW library services. On average the participants felt comfortable and familiar with everyday computing activities (mean 4.69, SD 0.63) and using web pages for on-line activities (mean 4.46, SD 0.88). However, participants were only mildly familiar or comfortable in the design and creation of web pages (mean 3.15, SD 0.69), with UW library web pages (mean 3.15, SD 0.90), and UW library services (mean 3.08, SD 0.95).
Apparatus
The lab-based usability testing was completed in the Engineering Lecture Hall computer laboratory. The users were studied as they interacted with a functional medium fidelity prototype loaded into Internet Explorer. The experimenters had a script to follow, which included the ten tasks for the users to answer.
Method
Prior to participation in the study, users were required to read and sign an information consent letter, which is reproduced in Appendix E. Subsequently, a background questionnaire was administered to users to collect general information on user demographics and computer and library experience (please refer to Appendix F for a reproduction of the questionnaire).
Participants were informed that participation in the study would require approximately 20-30 minutes of their time. They were instructed that participation was completely voluntary and that they could withdraw from the study at any time. Furthermore, participants’ identity remained confidential by assigning each with a number (e.g. P1).
Members of the design team acted as test monitors and followed the procedure as outlined in a script found in Appendix G. They instructed users to position their hand on the mouse and to move the pointer to the bottom of the center of the screen. Subsequently, the users were asked to perform the set of 10 tasks related to library activities using the redesigned Gateway page. The sequence in which each user completed the ten questions were alternated between sequence A and sequence B. This was an attempt to control for the learning curve users may experience as they became accustomed to the gateway. As the user performed the assigned tasks, the time taken to complete the task as well as any difficulties encountered in completing the task were recorded in a table as shown in Appendix A.
Following completion of the work tasks, users were asked to fill out a Usability Questionnaire, which is reproduced in Appendix H. Furthermore, each user was provided with a feedback letter as found in Appendix I.
Statistical Analysis
The actual times (mean values) to complete the tasks (as recorded during the lab-based usability testing) were compared to the theoretical times determined by keystroke analyses using t-test statistical methods. The results from the usability questionnaire were evaluated using a chi-square statistical method of analysis. For sample calculations for the statistical procedures, please refer to Appendix J.
Response Times
There were significant differences between the mean values for actually answering the usability questions (as determined by lab-based usability testing) and the theoretical answer times (as determined by the keystroke analyses) for many of the questions (please refer to Table 1 for significance values). It followed that many of the actual times were significantly longer than the theoretical values.
The differences in response times for three questions (actual vs. keystroke analyses values) were insignificant. This suggests that the users were able to quickly answer the questions on the same level the "expert" assumed with the keystroke analyses. The questions that were done well included: What are the hours for the university archives?, What books are available at the Kitchener Public Library?, and Does the UW library have a copy of Andrew Pyper’s Lost Girls? These questions all appear to be straight forward as the answers for the first two questions were direct links on the gateway (e.g. library hours, Kitchener Public Library). The answer to the third question was to use the "TRELLIS: Library Catalogue" link either on the navigation bar or in the "Books, Journals, Databases" menu. The quick response to this question is attributed to users previously using this function. In general, the quick response times to answer these three questions are attributed to the intuitive semantics and location of the link within the gateway.
The differences in response times for three questions (actual vs. keystroke analyses values) were insignificant. This suggests that the users were able to quickly answer the questions on the same level the "expert" assumed with the keystroke analyses. The questions that were done well included: What are the hours for the university archives?, What books are available at the Kitchener Public Library?, and Does the UW library have a copy of Andrew Pyper’s Lost Girls? These questions all appear to be straight forward as the answers for the first two questions were direct links on the gateway (e.g. library hours, Kitchener Public Library). The answer to the third question was to use the "TRELLIS: Library Catalogue" link either on the navigation bar or in the "Books, Journals, Databases" menu. The quick response to this question is attributed to users previously using this function. In general, the quick response times to answer these three questions are attributed to the intuitive semantics and location of the link within the gateway.
Table 1: Significant differences between the actual and theoretical (keystroke analyses) values for each question asked during the lab-based testing.
Question |
Mean Actual Time (s) |
Theor. Time * (s) |
Sig. Diff. between Actual and Theoretical |
1. Where do you go to find help with a research topic in your department? |
24.80 |
5.70 |
yes (tx (df=12) = 3.772, p < 0.05) |
2. How do you get an article from a journal that is available at WLU? |
16.89 |
5.30 |
yes (tx(df=12) = 3.075, p < 0.05) |
3. Can Alumni borrow books from UW? |
31.29 |
3.95 |
yes (tx(df=12) = 3.395, p < 0.05) |
4. What are the hours for the University Archives? |
10.42 |
5.45 |
no (tx(df=12) = 2.042, p > 0.05) |
5. What books are available at the Kitchener Public Library? |
10.19 |
5.30 |
no (tx(df=12) = 1.493, p >0.05) |
6. Does the library have any electronic dictionaries? |
19.06 |
5.30 |
yes (tx(df=12) = 2.874, p < 0.05) |
7. Does the UW library have a copy of Andrew Pyper’s Lost Girls? |
6.90 |
5.30 |
no (tx(df=12) = 1.386, p > 0.05) |
8. Where can you find instructions for connecting from home? |
41.61 |
5.30 |
yes (tx(df=12) = 3.929, p < 0.05) |
9. Where would you go to renew books online? |
11.70 |
5.30 |
yes (tx(df=12) = 3.103, p < 0.05) |
10. What is the contact information for the Liaison Librarian assigned to your home department? |
27.41 |
5.30 |
yes (tx(df=12) = 3.505, p < 0.05) |
* Theoretical times, as determined by keystroke level analysis, have not been adjusted to include reading time (2 words per second).
The users took longer to answer the majority of the questions when compared to the times proposed by the keystroke analyses (i.e. differences were significant). These questions are considered in detail below. As a general comment however, the discrepancies may not solely be attributed to usability issues of the gateway. Instead some of the assumptions made for the keystroke analyses and the testing procedures may have influenced the timed results. For example, the investigators may have timed until the user actually answered the goal, where as the keystroke analyses only timed until the users got off the gateway. As well, in the keystroke level analyses it was assumed that the users were at least experts. In fact many of the participants were unfamiliar with the site, the current UW library gateway, and/or the services provided by the UW library. Therefore some of the users would be better classified as novice users, and as such would take longer to complete the tasks. As well, reading time was not accounted for because the design team felt that the mental preparation would account for time to scan. It was also considered unjust to assume that the users read over the entire site, top to bottom / left to right (until they reached the link of choice), each time they carried out a task. The slow answering times may also be attributed to the users expecting a pop-up menu to automatically appear to the right of the gateway when they moused over the major menu headings. The users waited for the menu to appear, when in fact they need to click on the menu heading for the contents to appear on the screen. This waiting would increase the amount of time spent in answering the questions.
Question 1: Where would you go to find general help with a research topic in your department?
Answering this question took a significantly longer time (mean of 24.80 sec) than the theoretical answering time proposed by the keystroke analyses (5.70 sec). Analyzing user comments and observations made by the experimenters provided some explanation for the discrepancy. For example, most users did not know which option to choose. They ended up clicking on all of the major headings and reading over the associated menu items. Many users chose to look under "Undergraduates" or "Faculty and Staff" under the "Resource for UW People" menu. As a last resort the users often turned to the "How Do I?" or "Search the Library Site" functions. One user also turned to the "My Account Info" heading, in part because it sounded personal to them, and thus they thought they would also find information about their department on the menu. The correct link is called "Research Guides by Subject" under the "Online Reference Tools" main heading. "Research Guides by Subject" was not associated with an online tool and thus putting this link under the "Online Reference Tools" menu was not an appropriate match to user mental models. However, since many of the users clicked on each major heading, they would have read the "Research Guides by Subject" link. Even still the users did not associate the link with getting help on a research topic by department. Therefore, the semantics of the link do not adequately convey the services provided by the link. Given that most of the users looked under the "Resources for UW People" it would seem appropriate to put "Research Guides by Subject" under that category. As well, a new semantic should be considered for this link. Making these changes should help to increase the speed at which users could answer this question. As a general note however, some users remarked how the question was confusing which made it difficult to find an answer.
Question 2: How do you get an article from a journal that is available at the WLU?
There are many ways to answer this question: using TUGdocs, or going to the WLU website, the "Get from Guelph/Laurier/Annex" link or the "Get from Other Libraries" link under the "Other Libraries" menu. Most users were able to answer the question relatively quickly. Few users used the TUGdoc menu, suggesting that they did not know what the service is, or that they did not see the option. Since there are three different choices on the "Other Libraries" menu however, there may have been too many options to choose from to make, thus slowing the users down.
Question 3: Can Alumni borrow books from UW?
Surprisingly this took users more time to answer than the time proposed through keystroke analyses. Many users first went to the "Library and Community Info" menu as their first choice, perhaps on the idea that this menu would contain information about who may and may not borrow resources from the library. Other users were reported to do "lots of wandering". Eventually users used the "Search the Library Site" or the "How Do I?" functions available on the gateway. There were users who did chose "Resource for UW People" (correct response) as their first choice, however, it took them a while to make that decision. These findings suggest that the users were looking more for information regarding the rules for borrowing items from the library more than the type of person in question. It may be that the wording of the question did not suggest (or it was not intuitive) to look for information about Alumni and therefore users did not look under "Resources for UW People."
Question 6: Does the library have any electronic dictionaries?
Electronic dictionaries are found under the "Reference Tools" link from the "Online Reference Tools" menu. Although the designers had given this resource a great deal of consideration, the change in semantics of the "Reference Tools" link did not enhance match between the system and user mental model. The users took longer to locate the electronic dictionaries than the hypothesized time from the keystroke analyses. Many users noted that they did not even know electronic dictionaries existed, however, they were able to correctly choose the correct major menu (i.e. "Online Reference Tools"). Most of the difficulty arose in deciding the correct link from this menu, with the debate being between ‘Reference Tools’ and ‘Internet Search Tools’. It appears that it took the users awhile to attain their goal because the choice between the two link options was not intuitive enough. Interestingly, none of the users used the "Online Dictionaries" link that was featured under the "Highlight of the Week" menu to the left-hand side of the gateway. This suggests that the users are focusing their attention primarily on the middle section of the gateway. Perhaps then the left-hand side items are not as salient as necessary.
Question 8: Where can you find instructions for connecting from home?
Finding a way to connect from home appeared to be a problem for the users, as this question on average took the longest to answer (mean of 41.61 sec.) and was the most significantly different from the theoretically proposed answering time (tx(df=12) = 3.929, p < 0.05). Users did not know where to look and there was little consistency between each users attempt to locate the answer. A few users looked under the "Online Reference Tools" menu. Intuitively this would make sense seeing as connecting from home would involve an "online" approach. Hence the title of the menu, "Online Reference Tools" invited the users to begin their search with that menu. Alternatively some users chose to looked under "Resource for UW People" or "My Account Info" and eventually resorted to the "How Do I?" drop down menu. It appeared that the users were looking for a direct link that said something to the effect of "Connecting from Home". One user correctly resorted to using "e-journals" under the "Books, Journals, and Databases" menu. This method however, is a hidden or discrete way to connect from home and would require the user to follow a train of thought similar to: which resources would require me to connect from home à e-journals à go to that menu. For those users who do not use e-journals, they would not instinctively know to look under that link.
Question 9: Where would you go to renew books online?
Finding a way to renew books on line was answered relatively quickly (mean 11.70 secs) when compared to the other response times. Many of the users chose the right path (i.e. My Account Info à Renewals) however it just took time to decide to use that path to achieve the goal. There were however users that decided first to go to "Online Reference Tools". The keyword in completing the task was "renew" however, it may be that the users focused more on the "online" aspect of the question instead and hence chose to use the "Online Reference Tools" menu.
Question 10: What is the contact information for the Liaison Librarian assigned to your home department?
There are actually two different ways to answer this question, yet even with the flexibility users still took a significantly longer time to answer the question compared to the time proposed by the keystroke analysis. Most users resorted to the "Resources for UW People" as their first choice, and then chose the "Faculty and Staff" link. There were however users who were not sure of where to go, and searched throughout the gateway before coming upon an answer. It may be that many users do not know that liaison librarians exist or that this resource is not a direct link from the gateway.
Keywords
There were no statistically significant relationships between the levels of user-friendliness and thus the keywords on the gateway are equally user-friendly as they are neutral (impartial to user friendliness) and non-user friendly.
Table 2: Relation ship between User-friendly, Neutral, and Non-user Friendly for different usability characteristics on the proposed gateway.
Usability Item |
Significantly Different than Expected |
Keywords |
No (c 2observed (df=2) = 4.76, p > 0.05) |
Major Links |
No (c 2observed (df=2) = 5.70, p > 0.05) |
Graphics |
Yes (c 2observed (df=2) = 15.86, p < 0.05)User-friendly vs. Non-user Friendly: Yes (c 2observed (df=2) = 14.60, p < 0.05) |
Navigation Bars |
No (c 2observed (df=2) = 5.70, p < 0.05) |
Overall Layout |
Yes (c 2observed (df=2) = 12.16, p < 0.05)User-friendly vs. Non-user Friendly: Yes (c 2observed (df=2) = 11.75, p < 0.05) |
Major Links
Similar to keywords, the users found the major links on the gateway to be equally user-friendly as they were neutral and non-user friendly. This idea was reinforced when the users commented about their misconception of the main menu headings. Many of the users expected the main menu headings to act as "roll-overs" where a menu would appear to the right with the contents from which they could choose an action. In fact, the users were required to click the mouse on the main menu headings to see the contents. The users also mentioned that the semantics of the meaning of the menu headings were not always intuitive; that the heading name didn’t always portray an accurate message about its contents.
Graphics
There was a significant difference between the expected response and the observed response towards the graphics used on the proposed gateway. More users found the graphics to be user-friendly than non-user friendly or neutral (impartial).
Navigation Bar
Equal number of users found the navigation bar to be user-friendly, neutral (impartial) and non-user friendly.
Overall Layout
Significantly more users found the overall layout of the gateway to be user-friendly compared to those who were impartial (neutral) or found the overall layout to be non-user friendly. This significant relationship continued to hold true even when only the user-friendly and non-user friendly frequencies were considered.
Positives
Most of the users commented that the prototype was overall easy to use with a good layout and clear and uncluttered design. The users also felt that it was an attractive site, with a professional look. Most users felt the graphics were appropriate for the site. The prototype allowed for clear actions and useful feedback in regards to those actions. One user commented that they liked the navigation bar, yet they felt it would be more useful for frequent users. Another user enjoyed the web-site, but commented how it was necessary to "look around a bit" when first visiting the site. The central menu selection was also considered to work very well.
Dislikes and Troubles
Users commented that it was difficult to compare the prototype to the current site after a short period of testing. A third of the users mentioned that they expected rollover information on the right part of the screen, especially since the top text changes when mousing over menu headings and links. Other users found that the major links didn’t always tell them very effectively what sub-links could be expected on the menu. In relation to this point, some users noticed that certain links and functions were hidden or buried within the site (e.g. Connect from Home). There was also some confusion with multiple uses of the word "library". A user also questioned why "Community Info" was not linked with "Other libraries" instead of in connection with library information.
Recommendations
The users provided a variety of recommendations to improve upon the UW library gateway prototype and included the following:
Table 3: Rating of proposed gateway relative to current UW gateway
Rating |
% of users |
Not Applicable |
38.46 |
Much Better than Current |
15.38 |
Marginally Better than Current |
7.69 |
About the Same as Current |
30.77 |
Marginally Worse than Current |
7.69 |
Much Worse than Current |
0.00 |
Total |
100.00 |
7.3 Design Implications According to Results from Lab-based Testing
Semantics
The "Online Reference Tools" menu appeared to confuse the users, and thus the title will be altered to read "Research Tools". This change is also due in part to each item within the menu being related to different ways to carry out research. Within this menu heading, the link titled "Research Guides by Subject" will be changed to "Research Guides by Faculty". The word "Subject" was changed to "Faculty" so users will understand that they can search for information according to their department, which is actually the service the web-page provides. Similarly, the menu heading "Resources for UW People" will be modified by changing "Resources" to "Services" since the web-page provides services for the different students and staff at the University of Waterloo. This change was made in the hopes that more users will connect functions such as Librarian Liaisons, and information about Alumni to this menu.
Additions
Users searched the gateway to locate a direct link to connect their home computer to the resources available from the UW library gateway. Since such a link was missing from the proposed design, a "Connect from Home" link was added under the menu heading "Services for UW People".
Navigation and Side Menus
Most users did not notice (or at least had a difficult time finding) or use the navigation bar found underneath the Library and graphics header. To increase the saliency of the navigation bar, a larger space was inserted between the bar and the header. The font sizing was also enlarged to make the navigation bar even more noticeable. Users also appeared to ignore the side menus, such as the "Highlight of the Week". This feature is considered important because it introduces users to resources that they may not normally discover or use. To make the "Highlight of the Week" more prominent the background was changed to a pale yellow.
Rollovers vs. Clicking
A third of the users commented that they had expected menus to appear when they moused over the main menu headings. In fact, a few of these users mentioned their preference for menu rollovers rather than having to click on the heading to view the menu contents. The team felt that because the participants have used the current library gateway with roll-over menus, they may be partial to and thus expected the rollover mechanism. In reality there is a roll-over effect when the mouse is dragged over links on the prototype. Instead of menus appearing, an explanation of the contents of the link appears just below the navigation bar. Therefore, after a lengthy discussion, the design team decided to leave the menus as "clicks" rather than implementing rollovers.
Figure 5: Proposed UW Library Gateway Web Page
The physical specifications can be classified as those that are involved in the physical appearance and physical use of the site. They include the use of a screen resolution of 800x600, compliance to the UW visual identity standards, and the use of html and javascript code thus requiring a java enabled web browser.
The most prominent addition to the new gateway page was the left-hand side content area. This area included boxes for the ‘Search the Library Site’, ‘Highlight of the Week’, ‘Text-only Version’, and ‘Library Forms’. In the existing site, there was no evidence of a site index hence our recommendation for the ‘Search the Library Site’ feature. Based on the results of the discount usability method, this feature was found to be very useful not only for the novice user who can’t find what they are looking for on the site, but also for the experienced user who knows exactly where to go and wants to bypass the selection of the different headings. Thus the addition of such a tool is justified. Furthermore, the addition of the ‘Highlight of the Week’ which is also not present in the current site offers the user added insight into the many useful tools that are included in the present site that they are most likely not aware of. This feature also allows the library to promote tools that may help the user access information more efficiently. The addition of the ‘Text only version’ to this left-hand side content area was done in order to make this feature of the site more salient to the user. Results from user interviews and heuristic evaluations, this feature was not clear to most users in the existing gateway, in fact many users did not know that this feature existed. Though it is a feature that is not frequently used, the users that do need to use it also need to be able to find it quickly. Thus there is a relative importance of the text version function which justifies the addition of the feature into an area that makes it more salient. Finally, the ‘Library Forms’ section was found to very important through an interview with our client. This tool, though reported as one that is infrequently used by our participants, was reported by our client to be a feature that must be readily available on the gateway page. In order for the user to readily access this feature, it was necessary to include it on the gateway page as opposed to within a heading. These additions to the left-hand side content area are collectively justified due to their nature of importance. Though they are infrequently used, each element of this content area was reported to be necessary and thus required easy access, which is not currently offered in the current UW library gateway.
The removal of the graphic/watermark was also a key recommendation as per user interviews and the design team’s heuristic evaluations. The use of a white background as opposed to the existing watermark offers greater contrast that allows users to better differentiate the features of the site. The watermark of the existing site does not allow the user to readily distinguish usable features and thus decreases the efficiency of use. For this reason, a white background was chosen for the proposed gateway.
There were substantial recommendations to allow for more feedback from the buttons in the site as per the design team’s heuristic evaluations and user interviews. Feedback was enhanced in the main headings of the center content area and the sub headings located within them. First and foremost, the main headings were designed to offer a clear and distinct change in color upon mouse over in order to feedback to the user that it is a selectable link or button. Furthermore, upon the mouse over of the sub headings, there is also a clear and distinct color change of the background immediately surrounding the text of the sub heading. This is also to feedback to the user that they are selectable links to access information within them. Finally efforts were made to increase the feedback from the buttons on the top navigation bar by means of an appearance of a contrasting background color. These recommendations for feedback are either neglected in the existing site or poorly designed when present. It is clear that offering feedback to the user allows for more efficient access and visibility of the main features of the site and thus allows the user to more readily access information in the site.
Also related to the lack of feedback from the site, there is an addition of a text area that provides the user with a description of the contents found within each main heading upon mouse over. This addition was a direct result of the competitive analysis where it was reported that such a tool offered more efficiency when searching for information and allowed the user to preview the contents in order to prevent the user from entering the link in err.
As a result of the card sorting method, there were extensive changes in the language used for each of the main headings. Furthermore, there was a resulting rearrangement of the contents within the headings in order to offer a better match between the contents and their associated headings. These changes are justified by the lack of ability of many users in the design walkthrough to find specified contents within the main headings. Thus, the contents were rearranged and the headings renamed so that the user can retrieve information from the site more efficiently. Furthermore, the addition of the text ‘Online Catalog’ to the ‘Trellis’ link on the top navigation bar was found to better reflect what the link meant. For novice users of the library gateway, it was found that the link ‘Trellis’ had little meaning and by adding descriptive text to it, the link better represented the UW library’s catalog. All these changes collectively represent the enhancement of the semantics of the existing gateway page.
Another specification of the proposed gateway is the ‘How Do I?’ drop-down menu item in the top navigation bar. A similar help tool is on the existing gateway but was found in the design walkthroughs and the heuristic evaluations to be intimidating and not as helpful as it was meant to be. It was felt that a drop-down menu was required that offered not only the existing Trellis help section, but also features that are not frequently used and are difficult to find in the gateway. This feature thus helps novice users find what they are looking for quickly and efficiently.
There are also two additional recommendations that the design team has not been able to implement due to the limiting scope of the project. These are to create a web portal that is specific to each user’s preferences and to implement a consistent layout not only within the links directly related to the UW library, but also to the links that are related to the University itself. The creation of a web portal would allow users to include only the tools they use most frequently in the UW gateway. The design team felt that this feature would best serve all users and eliminate usability issues regarding the content of the gateway page. Furthermore, by allowing the individual user to customize the gateway to meet their own specifications, the user has complete control over their efficiency with the use of the site.
A consistent design of the layout for library related links and University related links would eliminate the user from getting reacquainted with each new page entered. By offering a consistent layout for each page, the user can locate information within each page more quickly. More specifically, the main navigation menu at the top of the proposed gateway is recommended to stay consistent within all library related pages as it was designed to allow the user access to the help menu, the library home page, and the Trellis library catalog. These were all found to be frequently used or needed when viewing the library related pages.
Preece, Jenny. (1993). A Guide to Usability: Human Factors in Computing. United States: Addison-Wesley Publishing Company.
Nielsen, J., Heuristics for User Interface Design, http://www.useit.com/papers/heuristic/heuristic_list.html, 2001, March 7, 2001.
Nielsen, J., and Mack, R. (1994). Usability inspection methods. New York: John Wiley & Sons.
Newman, W.M. and Lamming, M.G. (1995). Interactive System Design. United States: Addison-Wesley Publishing Company.
Norman, D. (1988). The Design of Everyday Things. New York: Currency Double Day.
Smith, K.A. (1998). Project Management and Teamwork. Boston: McGraw Hill.
Stanton, Neville. (1998). Human Factors in Consumer Products. United States: Taylor & Francis.
A – Data Collection Sheet and Usability Questions
B – Performance Times for Keystroke-level Operators
C – Sample Keystroke Level Analysis
D – Client Results tested with the Sequence A Script
E – Information / Consent Form
I – Feedback Letter
J – Sample Statistical Analyses
K – Lab-based Testing Results and Analyses
L – Summary of Client Meetings
N – Team Learning
APPENDIX A - Data Collection Sheet and Usability Questions
UW LIBRARY GATEWAY WEB PAGE
DATA COLLECTION SHEET
Test Monitor: ________________ Participant #: ________________
RECORD TIME AND OBSERVATIONS
TASK |
TIME (seconds) |
COMMENTS |
1 |
||
2 |
||
3 |
||
4 |
||
5 |
||
6 |
||
7 |
||
8 |
||
9 |
||
10 |
APPENDIX A (Con’t) - Data Collection Task List and Usability Questions
Sequence A |
|
Task |
Objective: Find the links that would allow you to answer the following ... |
1 |
Where would you go to find general help with a research topic in your department? |
2 |
How do you get an article from a journal that is available at WLU? |
3 |
Can alumni borrow books from UW? |
4 |
What are the hours for the University Archives? |
5 |
What books are available at the Kitchener Public Library? |
6 |
Does the library have any electronic dictionaries? |
7 |
Does UW library have a copy of Andrew Pyper’s Lost Girls? |
8 |
Where can you find instructions for connecting from home? |
9 |
Where would you go to renew books online? |
10 |
What is the contact information for the Liaison Librarian assigned to your home department? |
Sequence B |
|
Task |
Objective: Find the links that would allow you to answer the following ... |
1 |
What is the contact information for the Liaison Librarian assigned to your home department? |
2 |
Where would you go to renew books online? |
3 |
Where can you find instructions for connecting from home? |
4 |
Does UW library have a copy of Andrew Pyper’s Lost Girls? |
5 |
Does the library have any electronic dictionaries? |
6 |
What books are available at the Kitchener Public Library? |
7 |
What are the hours for the University Archives? |
8 |
Can alumni borrow books from UW? |
9 |
How do you get an article from a journal that is available at WLU? |
10 |
Where would you go to find general help with a research topic in your department? |
APPENDIX B –Performance Times for Keystroke-level Operators
APPENDIX C – Sample Keystroke Level Analysis
Using the "Search the Library Site"
STEP |
TIME (s) |
|
Mentally Prepare (Scan options) |
M |
1.35 |
Point mouse to dialogue box for "Search the Library Site" |
P |
1.10 |
Click on the dialogue box for "Search the Library Site" |
K |
0.20 |
Home hands on the keyboard |
H |
0.40 |
Type in Entry |
K |
0.20 x no. of keystrokes |
Follow either Sequence A, B, or C |
||
Sequence A - Home hands on the mouse |
H |
0.20 |
Point mouse to the "Search" button |
P |
1.10 |
Click the mouse on the "Search" button |
K |
0.20 |
Sequence B – Hit Enter Key |
K |
0.20 |
Sequence C – Hit Tab Key, then Enter Key |
K |
0.20 x 2 |
TOTAL: |
A = 4.55 + 0.20(no. of keystrokes)B = 3.25 + 0.20(no. of keystrokes) C = 3.45 + 0.20(no. of keystrokes) |
Using the "How Do I?"
STEP |
TIME (s) |
|
Mentally Prepare (Scan options) |
M |
1.35 |
Point mouse to drop down menu for "How Do I?" |
P |
1.10 |
Click on the drop down menu for "How Do I?" |
K |
0.20 |
Mentally Prepare (Scan menu options) |
M |
1.35 |
Point mouse to menu item of choice |
P |
1.10 |
Click on the menu item of choice |
K |
0.20 |
TOTAL: |
5.30 |
The following tables are the results of the keystroke analyses carried out for each task to be used in the lab-based usability testing:
Q1. Where would you go to find general help with a research topic in your department?
STEP |
TIME (s) |
|
Mentally Prepare (Scan options) |
M |
1.35 |
Point mouse to "Resources for UW People" OR Point mouse to "On-line Reference Tools" |
P |
1.10 |
|
K |
0.20 |
Mentally Prepare (Read menu options) |
M |
1.35 |
|
P |
1.10 |
|
K |
0.20 |
Off the gateway at this point |
||
GRAND TOTAL |
5.70 |
Q2. How do you get an article from a journal that is available at WLU?
OR
STEP |
TIME (s) |
|
Mentally Prepare (Scan options) |
M |
1.35 |
Point mouse to "Other Libraries" |
P |
1.10 |
Click on "Other Libraries" |
K |
0.20 |
Mentally Prepare (Read menu options) |
M |
1.35 |
|
P |
1.10 |
|
K |
0.20 |
Off the gateway at this point |
||
GRAND TOTAL |
5.30 |
STEP |
TIME (s) |
|
Mentally Prepare (Scan the site) |
M |
1.35 |
Point mouse to "Library Forms" pull down menu |
P |
1.10 |
Click on the drop down menu for "Library Forms" |
K |
0.20 |
Mentally Prepare (Scan list for item of choice) |
M |
1.35 |
Point mouse to "TUGdocs" |
P |
1.10 |
Click mouse on "TUGdocs" |
K |
0.20 |
GRAND TOTAL |
5.30 |
Q3. Can Alumni borrow books from UW?
STEP |
TIME (s) |
|
Mentally Prepare (Scan options) |
M |
1.35 |
Point mouse to "Resources for UW People" |
P |
1.10 |
Click on "Resources for UW People" |
K |
0.20 |
Mentally Prepare (Read over options) |
M |
1.35 |
Point mouse to "Alumni" |
P |
1.10 |
Click on "Alumni" |
K |
0.20 |
GRAND TOTAL |
5.30 |
Q4. What are the hours for the University Archives?
STEP |
TIME (s) |
|
Mentally prepare (scan the gateway) |
M |
1.35 |
Point mouse ‘Library & Community Info’ button |
P |
1.10 |
Click mouse button on ‘Library & Community Info’ |
K |
0.20 |
Point mouse ‘Library Hours’ |
P |
1.10 |
Click mouse button on ‘Library Hours’ |
K |
0.20 |
Off the gateway at this point TOTAL |
3.95 |
|
Point mouse ‘University Archives’ |
P |
1.10 |
Click mouse button on ‘University Archives’ |
K |
0.20 |
GRAND TOTAL |
5.25 |
Q5. What books are available at the Kitchener Public Library?
Choose heading appropriate for question |
M |
1.35 s |
Mouse over ‘Other Libraries’ |
P |
1.10 s |
Click on ‘Other Libraries’ |
K |
0.20 s |
Choose a sub-heading |
M |
1.35 s |
Mouse over ‘Kitchener Public Library’ |
P |
1.10 s |
Click on ‘Kitchener Public Library’ |
K |
0.20 s |
Off the gateway at this point |
||
GRAND TOTAL |
5.45 s |
Q6: Does the library have any electronic dictionaries?
STEP |
||
Choose heading appropriate for question |
M |
1.35 s |
Point mouse over ‘on-line reference tools’ |
P |
1.10 s |
Click on ‘on-line reference tools’ |
K |
0.20 s |
Choose a sub-heading |
M |
1.35 s |
Mouse over ‘reference tools’ |
P |
1.10 s |
Click on ‘reference tools’ |
K |
0.20 s |
Off the gateway at this point |
||
GRAND TOTAL |
5.30 s |
Q7. Does UW library have a copy of Andrew Pyper’s Lost Girls?
STEP |
TIME (s) |
|
Mentally Prepare (Scan gateway options) |
M |
1.35 |
Point to ‘TRELLIS Library Catalogue’ link on top navigation bar |
P |
1.10 |
Click mouse button on to ‘TRELLIS Library Catalogue’ |
K |
0.20 |
Off the gateway at this point |
||
TOTAL |
2.65 |
OR
STEP |
TIME (s) |
|
Mentally Prepare (Scan gateway options) |
M |
1.35 |
Point mouse to "Books, Journals, Databases" |
P |
1.10 |
Click mouse button on to "Books, Journals, Databases" |
K |
0.20 |
Mentally Prepare (scan the menu options available) |
M |
1.35 |
Point mouse to "TRELLIS: Library Catalogue" |
P |
1.10 |
Click mouse on "TRELLIS: Library Catalogue" |
K |
0.20 |
Off the gateway at this point |
||
GRAND TOTAL |
5.30 |
Q8. Where can you find instructions for connecting from Home?
STEP |
TIME (s) |
|
Mentally Prepare (Scan gateway options) |
M |
1.35 |
Point mouse to "How Do I?" |
P |
1.10 |
Click mouse on the drop down menu for "How Do I?" |
K |
0.20 |
Mentally Prepare (Chose an option from the menu) |
M |
1.35 |
Point mouse to "Connect from home" |
P |
1.10 |
Click mouse on "Connect from home" |
K |
0.20 |
GRAND TOTAL |
5.30 |
OR
STEP |
TIME (s) |
|
Mentally Prepare (Scan gateway options) |
M |
1.35 |
Point mouse to "Books, Journals, Databases" |
P |
1.10 |
Click mouse "Books, Journals, Databases" |
K |
0.20 |
Mentally Prepare (Chose an option from the menu) |
M |
1.35 |
Point mouse to "e-journals" |
P |
1.10 |
Click mouse on "e-journals" |
K |
0.20 |
Off gateway at this point TOTAL |
5.30 |
|
Mentally Prepare (Find link on the e-journal page) |
M |
1.35 |
Point mouse to "Connect from Home" link |
P |
1.10 |
Click mouse on "Connect from Home" link |
K |
0.20 |
GRAND TOTAL |
7.95 |
Q9. Where would you go to renew books online?
STEP |
TIME (s) |
|
Mentally prepare (after hearing sentence) |
M |
1.35 |
Point mouse to "My Account Info" |
P |
1.10 |
Click mouse on "My Account Info" |
K |
0.20 |
Mentally prepare (as user reads the menu options) |
M |
1.35 |
Move mouse pointer to "Renewals" |
P |
1.10 |
Click on "Renewals" link |
K |
0.20 |
Off gateway at this point |
||
GRAND TOTAL |
5.30 |
Q10. What is the contact information for the Liaison Librarian assigned to your home department?
STEP |
TIME (s) |
|
Mentally Prepare (scan the gateway options) |
M |
1.35 |
|
P |
1.10 |
|
K |
0.20 |
Mentally Prepare (decide menu item of choice) |
M |
1.35 |
|
P |
1.10 |
|
K |
0.20 |
Off gateway at this point TOTAL |
5.30 |
|
Mentally Prepare (locate Library Liaisons on the webpage) |
M |
1.35 |
Move mouse to the "Library Liaison" link |
P |
1.10 |
Click on the "Library Liaison" link |
K |
0.20 |
GRAND TOTAL |
7.95 |
Appendix D - Client Results tested with the Sequence A Script
Question |
Answer Time (s) |
1. Where do you go to find help with a research topic in your department? |
27 |
2. How do you get an article from a journal that is available at WLU? |
36 |
3. Can Alumni borrow books from UW? |
22 |
4. What are the hours for the University Archives? |
3 |
5. What books are available at the Kitchener Public Library? |
14 |
6. Does the library have any electronic dictionaries? |
4 |
7. Does the UW library have a copy of Andrew Pyper’s Lost Girls? |
3 |
8. Where can you find instructions for connecting from home? |
3 |
9. Where would you go to renew books online? |
4 |
10. What is the contact information for the Liaison Librarian assigned to your home department? |
36 |
Comments:
The client noted how she did not see the "Weekly Special" to the left-hand side of the gateway page. She also missed the description of the menu items in the header (underneath the navigation bar) until half way through the procedure. The client requested that the "My Info" menu heading be changed to "My Account Info" since she interpreted the former name to infer a portal option. The "Ask Us/Tell Us" link from the current UW Library gateway was not present in the prototype the client tested. She ask that it be reinstated to the navigation bar since it is a highly used function. As well, the client requested that a link for "UW Daily Bulletin" be added to the "Library and Community Info" menu. The client also noted her difficulties with the semantics of "Online Reference Tools".
APPENDIX E – Information / Consent Letter and Form
APPENDIX F - Background Questionnaire
USABILITY TESTING OF UW LIBRARY GATEWAY WEB PAGE.
Test Monitor: _____________ Participant # _____________
BACKGROUND QUESTIONNAIRE
General Questions:
Gender: ___ Male ___ Female Age: ____
Program: ___________________ Year: ____ Co-op: ___ Yes ___ No
On a scale of 1 to 5, how familiar would you say you are with everyday computing activities (e.g. word processing, email)?
| 1 | 2 |
3 |
4 |
5 |
| not at all familiar |
very familiar |
|||
On a scale of 1 to 5, how familiar would you say you are with the use of web-pages for on-line activities (e.g. searching for information)?
| 1 | 2 |
3 |
4 |
5 |
| not at all familiar |
very familiar |
|||
On a scale of 1 to 5, how familiar would you say you are with the design and creation of web-pages?
| 1 | 2 |
3 |
4 |
5 |
| not at all familiar |
very familiar |
|||
On a scale of 1 to 5, how familiar would you say you are with UW's Library web-pages?
| 1 | 2 |
3 |
4 |
5 |
| not at all familiar |
very familiar |
|||
On a scale of 1 to 5, how familiar would you say you are with UW's Library services?
| 1 | 2 |
3 |
4 |
5 |
| not at all familiar |
very familiar |
|||
APPENDIX G - Test Monitor Scripts
SD 348
Winter-01 Team Project
TEST MONITOR SCRIPTS
Note to Test Monitors:Please read the text in Times-Roman (this font) out loud to each participant. The text in capitals (Arial) are instructions to you and should not be read out loud.
LIST OF USER-TESTING TASKS:
Instructions to Participant, Information/Consent & Background Questionnaire (5 minutes)
Library Tasks using current design or redeisgn (10-20 minutes)
Usability Questionnaire (5 minutes)
EQUIPMENT YOU NEED TO BRING TO TESTING SESSIONS:
testing scriptdata collection forms
a watch to time events
clipboard (or something to write on)
extra paper for making notes
Step 1: INSTRUCTIONS TO PARTICIPANTS.
NOTE: MAKE SURE THAT WINDOW FOR WEB PAGE IS MINIMIZE SO THAT PARTICIPANT CANNOT SEE IT UNTIL THE TRIALS ARE READY TO BEGINThank you for agreeing to participate in our study entitled AUsability Testing of the UW Library Gateway Web Page@.
This study is being carried out by the class members of SD 348 (User-Centred Design) as part of our course requirements. In order to make sure that all participants receive the same information, I am going to read to you from this script.
We will start by going over the information letter for this study. It will explain the objectives of this study and the tasks that we will ask you to perform.
GO OVER INFORMATION LETTER, AND ASK IF PARTICIPANT HAS ANY QUESTIONS?
ASK PARTICIPANT TO READ OVER AND SIGN CONSENT FORM.
Step 2: BACKGROUND QUESTIONNAIRE
Before we have you use the UW Library Gateway Web Page we would like to ask you some general questions about yourself and your computing experience.ASK STUDENT TO FILL IN THE BACKGROUND QUESTIONNAIRE.
Now for the main part of the study.
Step 3: LIBRARY TASKS
Before we start, I need to remind you that we are testing the usability of the web page design and not your ability to use the system. I will be taking notes as you work through the tasks and recording how long it takes someone to find the appropriate links and where people may have difficulties using the web page. This will help us to improve upon our design before our presentations next week.I will be asking you to work through a series of tasks that people normally associate with libraries. We will be going through each task one right after the other and then I will give you an opportunity to comment about the web page and tasks once they are all done.
For each task, you will start with your hand on the mouse and the cursor at the bottom centre of the screen. I will then read the task out loud and once I am finished you are to try to find the appropriate link on the web page as quickly as possible. I will tell you when you have found the appropriate link which will end that task. I will then remind you to move the cursor to the bottom centre of the screen before I read the next task.
- BRING UP WINDOW AND ASK PARTICIPANT TO PLACE HAND ON MOUSE AND MOVE CURSOR TO BOTTOM CENTRE OF SCREEN.
- READ FIRST TASK FROM SHEET.
- START TIMER.
- STOP TIMER WHEN PARTICIPANT HAS CLICKED ON APPROPRIATE LINK THAT MOVES USER OFF OF THE GATEWAY.
- NOTE ANY PROBLEMS ON OBSERVATION SHEET BEFORE MOVING TO NEXT TASK.
- REMIND PARTICIPANT TO MOVE CURSOR TO BOTTOM CENTRE AND KEEP HAND ON MOUSE WHILE YOU READ NEXT TASK OBJECTIVE.
Step 4: USABILITY QUESTIONNAIRE
As a final task we would like you to answer some questions about the usability of the UW Library Gateway Web-Page design that you just used.APPENDIX H - Usability Questionnaire
UW LIBRARY GATEWAY WEB PAGE
USABILITY QUESTIONNAIRE
Test Monitor: __________________ Participant # _______________
| Not at all User-friendly |
Very user- Friendly |
||||
| Key words | 1 |
2 |
3 |
4 |
5 |
| Major Links | 1 |
2 |
3 |
4 |
5 |
| Graphics | 1 |
2 |
3 |
4 |
5 |
| Navigation Bars | 1 |
2 |
3 |
4 |
5 |
| Overall Layout | 1 |
2 | 3 | 4 | 5 |
A) In terms of overall usability, how would you compare this web page with the current UW Library Gateway Page
| 1 | not applicable, (haven’t used current UW Library Gateway Page) | go to #3 |
| 2 | much better than current UW Library Gateway Page | go to #3 |
| 3 | marginally better than current UW Library Gateway Page | go to #3 |
| 4 | about the same as current UW Library Gateway Page | go to #3 |
| 5 | marginally worse than current UW Library Gateway Page | (go to #2b) |
| 6 | much worse than current UW Library Gateway Page | (go to #2b) |
B) In your opinion, what would you say are the major problems with the new design that makes it worse than the current current UW Library Gateway Page?
Do you have any additional comments or feedback on the web-pages you used today?
APPENDIX I - Feedback Letter
USABILITY OF THE UW LIBRARY GATEWAY WEB PAGE
March 22, 2001
Dear Participant:
Thank you for helping us with our usability study as part of our course requirements for SD 348 (User-Centred Design). The class has been working with the UW Library's Community Needs Assessment Group to help improve the usability of UW's Library Gateway Web Page. UW would like to make sure that its library resources and services are accessible and easy to find for UW students and faculty, UW alumni, and the larger community. Improving the usability of the library web pages will help them to meet that goal.
There are six different teams in our class. Each team has been working on an alternative design for the current UW Library Gateway Web Page. We will be using the information that we have gathered from you along with the other participants to help improve upon the usability of our web page designs. As stated in the information letter, the data from participants will be analysed as a group and you will not be identified personally.
As a class we have been following a user-centred approach to design. Each team has developed their own prototype of the UW Library Gateway that has evolved over the course of the term through usability analysis and testing. Our class will be giving a presentation of the evolution of our web designs on
Tuesday, March 27, 2001at 9 am - 10:30 am in E2 1303 B.
You are welcome to attend to see how your input may have helped with the development of the final designs.
If you would like further information about this project, please contact Prof. Carolyn MacGregor at 888-4567 ext 2897 or at cgmacgre@engmail.uwaterloo.ca
If you have concerns regarding your participation in this study, please contact Dr. Susan Sykes at the Office of Research Ethics, 888-4567 ext 6005.
Thank you again for your participation.
The Students of SD 348.
APPENDIX J - Sample Statistical Analyses
T-test Solutions
General Question:
Is the time to answer a question using the actual prototype different than the amount of time hypothesized from the keystroke analysis?
(two tailed test – not asking if one way is better than the other)
Example: For Test Question 1
| Sample size | = N = 13 |
| Required value (as determined by the keystroke analysis) | = R = 5.70 |
| Standard deviation | = s = 18.25 seconds |
| Alpha Level (level of significance) | = a = 0.05 (two tailed test) |
| Degrees of Freedom | (N-1) = df = 12 |
| t-critical (value above which t-observed must lie to be significant) | = tcrit = 2.179 |
Hypothesis:
H0: There is no difference between the actual and hypothesized answering times R = 5.70
H1: There is a difference between the actual and hypothesized answering times R = 5.70
Decision Rules:
Retain H0 if tobserved = 2.179
Reject H0 if tobserved ³ ½ 2.179½
Calculation:
Outcome: X = 24.80
Compute:
t observed = X outcome - R Ho
sx
= 24.80 – 5.70
5.06
= 3.772
Decision:
Reject Ho (since tobserved > 2.179)
There is a difference between the actual and hypothesized answering times. It takes longer to perform the task than the theoretically determined time (i.e. keystroke analyses).
(tx (df=12) = 3.772, p < 0.05)
*** NOTE: A similar conclusion is derived using confidence intervals.
Confidence Interval:
When the l is outside of the confidence interval than the null hypothesis is rejected.
C.I. = X +/- (tcritical x sx)
Compute:
C.I. max = 24.80 + (2.179 x 5.06) = 35.830
C.I. min = 24.80 – (2.179 x 5.06) = 13.766
Conclusion:
The probability is a 0.95 that m lies within an interval such as 13.77 to 35.83.
Since the m , required value of 5.70 does not lie within the range, the null hypothesis is rejected and thus there is a difference between the actual and hypothesized answering times. It is proposed that it took the participants longer to perform the task than the theoretically determined time (i.e. keystroke analyses).
Chi Squared Solutions
General Question:
Is there a relationship between keywords and the level of user-friendliness (as determined by the users)?
Please note that the categories of user-friendliness were defined as follows:
| Not at all User-friendly | Very user-Friendly |
||||
| Graphics | |||||
| Non user-friendly | Neutral |
Very user-friendly |
|||
Example: Graphics
| Sample size | = N = 13 |
| Categories (user friendly, neutral, non-user friendly) | = n = 3 |
| Alpha Level (level of significance) | = a = 0.05 |
| Degrees of Freedom (n-1) | = df = 2 |
| c 2-critical (value above which c 2-observed must lie to be significant) | = c 2crit = 5.9915 |
Hypothesis:
H0: There is no difference between the observed and expected values.
(i.e. the graphics were equally user-friendly, neutral, and non-user friendly)
H1: There is a difference between the observed and expected values.
Decision Rules:
Retain H0 if c 2observed < 5.9915
Reject H0 if c 2observed m 5.9915
Calculation:
c 2observed = S [(obs – exp)2/exp]
General Calculation
expected (N / no. of categories) |
observed (frequency) |
obs - exp |
(obs-exp)2 |
(obs-exp)2 exp |
|
user friendly |
4.33 |
11 |
6.67 |
44.49 |
10.27 |
neutral |
4.33 |
2 |
-2.33 |
5.43 |
1.25 |
non-user friendly |
4.33 |
0 |
-4.33 |
18.75 |
4.33 |
c 2observed |
15.86 |
General Decision:
Reject Ho (since c 2observed > 5.9915)
General Conclusion:
There is a relationship between keywords and the level of user-friendliness (as determined by the users), (c 2observed (df=2) = 15.86, p < 0.05).
Specific Comparisons:
Comparing User-friendly with Neutral
c 2observed = 10.27 + 1.25 = 11.52 (significant)
Comparing Neutral with Non-user friendly
c 2observed = 1.52 + 4.33 = 5.85 (in-significant)
Comparing Non-user friendly with User-friendly
c 2observed = 4.33 + 10.27 = 14.60 (significant)
Specific Conclusions:
There is a significant relationship between user-friendly and neutral accounts of the graphics used on the prototype (c 2observed (df=2) = 11.52, p < 0.05). It is inferred that more people found the graphics to be user-friendly than those who were impartial to the graphics.
There is no relationship between the neutral and non-user friendly accounts of the graphics found on the prototype (c 2observed (df=2) = 5.85, p > 0.05).
There is a significant relationship between user-friendly and non user-friendly accounts of the graphics on the gateway prototype (c 2observed (df=2) = 14.60, p < 0.05). The results propose that more users found the graphics to be user-friendly than non-user friendly. This comparison provides the most valuable information.
APPENDIX K - Lab-based Testing Results and Analyses
General Information about Users/Participants
Questions |
|||||||||
Program |
Gender |
Age |
School Year |
1 |
2 |
3 |
4 |
5 |
|
1 |
Engineering (Comp.) |
m |
19 |
2 |
5 |
5 |
4 |
3 |
3 |
2 |
Science and Business |
f |
23 |
4 |
3 |
2 |
2 |
2 |
2 |
3 |
Engineering (Sy. De.) |
f |
23 |
5 |
4 |
4 |
4 |
2 |
2 |
4 |
Kinesiology |
m |
21 |
3 |
5 |
5 |
3 |
2 |
2 |
5 |
Engineering (Sy. De.) |
f |
23 |
4 |
5 |
5 |
3 |
4 |
4 |
6 |
Science (Biol.) |
f |
19 |
2 |
4 |
4 |
3 |
3 |
3 |
7 |
Kinesiology |
f |
23 |
3 |
5 |
5 |
2 |
5 |
4 |
8 |
Computer Science |
f |
18 |
1 |
5 |
5 |
3 |
3 |
2 |
9 |
Arts (Psych, Econ) |
f |
23 |
4 |
5 |
5 |
3 |
4 |
5 |
10 |
Engineering (Sy. De.) |
m |
- |
m.sc. |
5 |
5 |
4 |
4 |
4 |
11 |
Kinesiology |
m |
21 |
3 |
5 |
4 |
3 |
3 |
3 |
12 |
Engineering (Sy. De.) |
m |
37 |
4 |
5 |
4 |
3 |
3 |
3 |
13 |
Engineering (Elec.) |
f |
20 |
2 |
5 |
5 |
4 |
3 |
3 |
mean |
5m, 8f |
22.5 |
3.08 |
4.69 |
4.46 |
3.15 |
3.15 |
3.08 |
|
SD |
4.93 |
1.16 |
0.63 |
0.88 |
0.69 |
0.90 |
0.95 |
||
Questions
Q1 - Familiarity with everyday computing activities
Q2 - Familiarity using web-pages for on-line activities (e.g. Searching for info)
Q3 - Familiarity with design and creation of web pages
Q4 - Familiarity with UW library web-pages
Q5 - Familiarity with UW library services
Question Rating Scale
1= not very familiar
5= very familiar
Results for Timed Tasks
Actual Sequence |
Timed results in A sequence (secs) |
|||||||||
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
|
B |
30.28 |
27.85 |
18.29 |
12.15 |
4.91 |
45.73 |
6.17 |
40.60 |
17.37 |
81.60 |
A |
36.05 |
16.05 |
65.73 |
12.99 |
9.01 |
49.72 |
13.16 |
70.94 |
15.64 |
20.34 |
B |
10.92 |
6.25 |
33.00 |
5.60 |
9.96 |
6.06 |
4.67 |
59.97 |
6.38 |
24.26 |
A |
20.72 |
4.00 |
2.66 |
4.87 |
6.78 |
42.00 |
2.71 |
122.23 |
3.16 |
52.18 |
A |
26.05 |
7.21 |
21.21 |
5.00 |
2.05 |
20.00 |
6.51 |
73.30 |
6.42 |
29.07 |
B |
15.91 |
12.15 |
76.87 |
11.75 |
46.54 |
3.95 |
6.41 |
14.62 |
8.36 |
49.18 |
A |
1.37 |
9.47 |
17.20 |
4.61 |
13.53 |
9.41 |
6.10 |
20.00 |
5.29 |
36.50 |
B |
10.00 |
44.00 |
78.00 |
3.00 |
3.00 |
4.00 |
4.00 |
3.00 |
4.00 |
8.00 |
A |
1.35 |
25.00 |
17.52 |
46.00 |
8.32 |
11.00 |
10.29 |
1.06 |
5.60 |
7.00 |
B |
29.07 |
8.53 |
5.70 |
4.51 |
4.93 |
8.36 |
4.58 |
31.24 |
26.77 |
12.14 |
B |
62.00 |
41.00 |
12.00 |
15.00 |
9.00 |
7.00 |
3.00 |
39.00 |
17.00 |
3.00 |
A |
25.00 |
15.00 |
5.00 |
4.00 |
11.00 |
7.00 |
17.00 |
35.00 |
17.00 |
7.00 |
A |
53.65 |
3.09 |
53.65 |
5.97 |
3.39 |
33.49 |
5.15 |
29.97 |
19.17 |
26.09 |
Mean |
24.80 |
16.89 |
31.29 |
10.42 |
10.19 |
19.06 |
6.90 |
41.61 |
11.70 |
27.41 |
SD |
18.25 |
13.59 |
27.61 |
11.43 |
11.44 |
17.26 |
4.17 |
33.32 |
7.44 |
22.75 |
Result for t-tests and Confidence Intervals between Timed Tasks and Keystroke Analyses
Timed Questions |
||||||||||
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
|
Mean (s) |
24.80 |
16.89 |
31.29 |
10.42 |
10.19 |
19.06 |
6.90 |
41.61 |
11.70 |
27.41 |
Req'd Value (s) |
5.70 |
5.30 |
5.30 |
3.95 |
5.45 |
5.30 |
5.30 |
5.30 |
5.30 |
5.30 |
Stand. Error of Mean |
5.06 |
3.77 |
7.66 |
3.17 |
3.17 |
4.79 |
1.16 |
9.24 |
2.06 |
6.31 |
t - observed |
3.772 |
3.075 |
3.395 |
2.042 |
1.493 |
2.874 |
1.386 |
3.929 |
3.103 |
3.505 |
alpha level (two tailed) |
0.05 |
0.05 |
0.05 |
0.05 |
0.05 |
0.05 |
0.05 |
0.05 |
0.05 |
0.05 |
t - critical |
2.179 |
2.179 |
2.179 |
2.179 |
2.179 |
2.179 |
2.179 |
2.179 |
2.179 |
2.179 |
Significant? |
yes |
yes |
yes |
no |
no |
yes |
no |
yes |
yes |
yes |
Confidence Interval |
||||||||||
X max |
35.830 |
25.107 |
47.978 |
17.324 |
17.101 |
29.485 |
9.425 |
61.749 |
16.202 |
41.160 |
X min |
13.766 |
8.677 |
14.611 |
3.514 |
3.272 |
8.625 |
4.383 |
21.471 |
7.207 |
13.664 |
Score and Frequency values for Post-testing Usability Ratings
User |
Keywords** |
Major Links** |
Graphics** |
Navigation Bar** |
Overall Layout** |
1 |
4 |
4 |
4 |
5 |
4 |
2 |
- |
4 |
4 |
5 |
4 |
3 |
4 |
4 |
5 |
5 |
4 |
4 |
4 |
3 |
4 |
2 |
3 |
5 |
3 |
3 |
4 |
3 |
4 |
6 |
- |
4 |
5 |
5 |
5 |
7 |
- |
3 |
4 |
4 |
5 |
8 |
3 |
3 |
5 |
4 |
5 |
9 |
- |
3 |
4 |
3 |
4 |
10 |
- |
3 |
4 |
4 |
4 |
11 |
4 |
2 |
3 |
3 |
3 |
12 |
3 |
3 |
3 |
3 |
3 |
13 |
- |
3 |
5 |
5 |
4 |
mean |
3.57 |
3.23 |
4.15 |
3.92 |
4.00 |
SD |
0.53 |
0.60 |
0.69 |
1.04 |
0.71 |
frequencies |
|||||
user friendly( score 4-5) |
4 |
4 |
11 |
8 |
10 |
neutral (score 3) |
3 |
8 |
2 |
4 |
3 |
non-user friendly (score 1-2) |
0 |
1 |
0 |
1 |
0 |
Chi Square and Frequency values (post testing)
Keywords (n=7) |
|||||
expected |
observed |
obs - exp |
(obs-exp)2 |
(obs-exp)2/exp |
|
user friendly |
4.33 |
4 |
-0.33 |
0.11 |
0.03 |
neutral |
4.33 |
3 |
-1.33 |
1.77 |
0.41 |
non-user friendly |
4.33 |
0 |
-4.33 |
18.75 |
4.33 |
chi square |
4.76 |
||||
Alpha level |
0.05 |
||||
df = 2 |
|||||
Significant? |
no |
||||
Major Links (n=13) |
|||||
expected |
observed |
obs - exp |
(obs-exp)2 |
(obs-exp)2/exp |
|
user friendly |
4.33 |
4 |
-0.33 |
0.11 |
0.03 |
neutral |
4.33 |
8 |
3.67 |
13.47 |
3.11 |
non-user friendly |
4.33 |
1 |
-3.33 |
11.09 |
2.56 |
chi square |
5.70 |
||||
Alpha level |
0.05 |
||||
df = 2 |
|||||
Significant? |
no |
||||
Graphics (n=13) |
|||||
expected |
observed |
obs - exp |
(obs-exp)2 |
(obs-exp)2/exp |
|
user friendly |
4.33 |
11 |
6.67 |
44.49 |
10.27 |
neutral |
4.33 |
2 |
-2.33 |
5.43 |
1.25 |
non-user friendly |
4.33 |
0 |
-4.33 |
18.75 |
4.33 |
chi square |
15.86 |
||||
alpha level |
0.05 |
||||
df = 2 |
|||||
Significant? |
yes |
||||
Navigational Bars |
|||||
expected |
observed |
obs - exp |
(obs-exp)2 |
(obs-exp)2/exp |
|
user friendly |
4.33 |
8 |
3.67 |
13.47 |
3.11 |
neutral |
4.33 |
4 |
-0.33 |
0.11 |
0.03 |
non-user friendly |
4.33 |
1 |
-3.33 |
11.09 |
2.56 |
chi square |
5.70 |
||||
alpha level |
0.05 |
||||
df = 2 |
|||||
Significant? |
no |
||||
Overall Layout |
|||||
expected |
observed |
obs - exp |
(obs-exp)2 |
(obs-exp)2/exp |
|
user friendly |
4.33 |
10 |
5.67 |
32.15 |
7.42 |
neutral |
4.33 |
3 |
-1.33 |
1.77 |
0.41 |
non-user friendly |
4.33 |
0 |
-4.33 |
18.75 |
4.33 |
chi square |
12.16 |
||||
Alpha level |
0.05 |
||||
df = 2 |
|||||
Significant? |
yes |
||||
APPENDIX L – Summary of Client Meetings
Client Meeting #1
Date: January 25. 12:30pm.
Location: Library Seminar Room
Attendance: All. Ankur Gupta, Veronica Haliniak, Pamela Lauz, Barry Piquette, Julia Thompson,Simon Yu, and client Anne Fullerton.
In this meeting, the design team learned of the current UW Library Gateway history and of its problems as defined by Anne Fullerton. Using a rough heuristic evaluation, the team reported shortcomings and suggested some small changes to remedy some of the lacking areas. The project scope was identified as the Gateway page with secondary pages (those immediately accessed via the gateway) being changed at the group’s discretion. Commonality and consistency were stressed with the old and new designs. Future directions were revealed by the end of the meeting.
Meeting minutes are as follows:
Discussion:
- Website purpose.
Points and properties of the website:
- provide easy access to library functions (pointers to information, quick references).
- speed of finding material for new users.
- provide non-academic information to users.
- accessibility to new reference material (ex: $5.7M is spent annually to update material; needs to be accessed).
- System Boundaries.
What is the Gateway?
- The Top page of library website & (maybe) secondary pages.
- The graphical and text versions must be reviewed for those who use both pages. Text based page faster and used by telnetters.
- Consistency with rest of site. Navigation bar should be present and consistent throughout the site.
- Complex vs. simple? (Is less more? Do we need to change navigation bar?)
- Designed last year
- Old design was purely text-based.
- Commonly used articles were difficult to find.
- Usability testing WAS performed with 6 people. Faculty and students used verbal walkthroughs.
Future direction.
- Group will focus on Gateway, not individual portals.
- Group to consider more salient HELP section.
- Group to consider FAQ section.
- Group to consider Trellis access from anywhere within the site (perhaps beyond scope of project).
All members are to:
- bring personal suggestions for changes to Gateway to next meeting.
- look at other school websites. Perform heuristics evaluations.
- form list of likes/dislikes of other library Gateway pages.
Meeting adjourned, 1:40pmClient Meeting #2
Date: March 1st. 12:30pm.
Location: Library Seminar Room
Attendance: All. Ankur Gupta, Veronica Haliniak, Pamela Lauz, Barry Piquette, Julia Thompson, Simon Yu.
Client meeting cancelled due to illness. Regular meeting held.
Client meeting #2 scheduled for March 6th via e-mail.Client Meeting #2 (Rescheduled)
Date: March 6th. 1:30pm.
Location: Dana Porter Library Multimedia Room
Attendance: All. Ankur Gupta, Veronica Haliniak, Pamela Lauz, Barry Piquette, Julia Thompson, Simon Yu, and client Anne Fullerton.In this meeting, the group revealed the progress to the client. The card sorting technique and its testing results were explained to the client. Anne was asked to perform the card sorting technique so that an expert opinion could be garnered. The low fidelity prototype was shown and explained. Among the good points were the Help aids, while the group was cautioned about the use of a Quick Search on the front page. The minutes of the meeting are as follows:
Display low fidelity prototype for feedback from Anne.
Quick Search option perhaps irrelevant (Change to Site Search) ~ searching does not parse all available data (Trellis, and others).
Weekly Special – good idea. Use database of unknown library features to display random link.Group explained Card Sorting technique to Anne.
Asked Anne to perform card sorting. Though difficult (because of familiarity with current site), she was able to categorize features and provide more insight to feature functions.How do I? section discussed.
Plan is to place miscellaneous links in here. (For ex: From Laurier link)
"From Laurier" link is explained ~ a link that shows HOW to obtain material from Laurier. It is NOT a redundant function of the Laurier Library site.Miscellaneous points:
- Make sure to test "Site Search" in usability. Does a user know what they're searching? (Not the catalogue)
- Suggestions (many) made about layout and functionality of webpage features. Explanations vs. menus in white space was discussed.
- provide Anne with e-copy of interim reports [Julia]
- make sure important links have not been removed due to perception of redundancy [Pam]
- test "Site search" when doing usability. (Ask users about expectations) [All]
Meeting adjourned 3:30pm.The design team deemed a March 22nd meeting would not be needed if one were held earlier (March 21st)
Client Meeting #3
Date: March 21st.
Location: DC Library, Anne Fullerton’s office
Attendance: Veronica Haliniak, Pamela Lauz and client Anne Fullerton.In this meeting, the client, Anne Fullerton was asked to perform the usability test on the medium fidelity prototype. Results showed that a few usability issues weren't dealt with as thoroughly as intended and that certain important links were absent from the Gateway. The following are minutes from that meeting.
Pilot Test of Keystroke Level Analysis (KLA)
- run client through KLA on current functional prototype (note, completely functional except for site search function, which will be functional by next testing time). Using script handed out in class, Anne performed the entire test as it will be completed in class on Thur March 22. This includes the information and consent form, initial background questionnaire, as well as completing all tasks as scripted in version A. Veronica read the script and timed the client, while Pam recorded times and users errors and any additional comments.
- After completing the 10 pre-assigned task questions, the client completed a usability questionnaire, and was given the feedback sheet to read. We then discussed with Anne the comments she made on the usability questionnaire, as well as errors made while performing the tasks. This run-through was useful for us in that it enabled us to find errors in the procedures and handouts that need clarification before giving to others for testing (eg. on usability questionnaire, what are keywords vs. major links, or when are we allowed to ask users open-ended questions (not in the script)).
- These are some of the comments made, and any changes to be made as a result:
- Weekly special not noticed right away \ move this block up to second in list and increase title one shade so to stand out a little but not too much.
- Also title "weekly special" not great \ change to "Highlight of the week"
- My info: Anne thinks it will lead users to believe it is a "portal creator" \ Change title to "my library account"
- Anne showed us the response rate to "ask us/tell us" button (stats we should include in the report, stuff like 70+ emails in January alone!) \ include button in top toolbar
- Anne also showed concern for no "UW daily bulletin" link (apparently a widely used feature) \ add link under "Library & Community" link
- "Research Guides" a link under "on-line reference tools", not intuitive that sorted by subject \ change to "Research guides by subject
- Anne was given a copy of the current prototype for her reference, and confirmed that she will be in attendance during testing on March 22 (although may be a bit late).
Appendix M - Team Project AllocationsPrior to the first phase of usability testing, all team members were required to individually assess the current UWL gateway page as well as various other university library gateway pages. The findings were not only used in the competitive analysis section of the first interim report but also to develop initial design specifications for the low-fidelity prototype. Note that all team members actively participated in meeting sessions and were required to provide results from the usability testing they conducted. Furthermore, individual task allocation for each iteration of the design cycle is documented below.
Interim Report 1
Julia was responsible for composing the introduction as well as the heuristic evaluations of the assessment of the current design. The design walkthroughs were conducted and reported by Pam, while Veronica was responsible for the composition and results of the hierarchical task analyses. Simon amalgamated everybody’s competitive analyses for the report. Ankur was responsible for the section titled "Proposals for Change" while Pam designed the preliminary design screens according to these suggestions. Julia integrated each section of the report and everybody was provided with the opportunity to review the report prior to submission.
Interim Report 2
Similar to Interim Report 1, Julia composed the brief introduction for Interim Report 2. Conducting the card sorting tasks was an effort by all team members; Pam put together the kits and everyone else tested two to four users of a variety of background. Barry, Veronica, Ankur, Simon, and Julia compiled the card sorting results and Pam implemented the changes to the prototype. Julia completed the methodology and results as discussed in the report. Discount usability testing was conducted by Veronica, Barry, Simon, Julia and Ankur. More specifically, Simon, Barry and Ankur performed the heuristic evaluation, while Julia and Veronica conducted the design walkthoughs. Ankur and Veronica were responsible for the discount usability write-up. Barry composed the proposals for the medium fidelity prototype while Pam was responsible for implementing the changes. With the help of everyone on the team, Simon and Julia integrated the sections and proof-read the report.
Final Report
The final report required contributions from every design team member. Veronica and Ankur wrote the executive summary, while Barry and Veronica focused their efforts on the "introduction to the problem", the "revised interactive systems problem statement" and the "project constraints and requirements." Julia was primarily responsible for the sections titled "Phase 1 UCD Methods" and the data analysis/results for the keystroke and lab-based testing. Ankur completed the "Phase 2 UCD Methods" section as well as the methodology description for the keystroke analysis and lab-based testing. Simon composed the results sections for both Phase 1 and Phase 2 UCD method sections of this report. Simon was also responsible for completing the final design specifications. Barry put together the summary of the client meetings. Once again, Pam implemented the changes to the prototype. Everyone from the group included their summary of "team learning". Julia, Ankur, and Simon put out large efforts in proof-reading the report prior to submission.
Appendix N – Team learningAnkur
The UW library gateway project provided an opportunity to further apply the methods and techniques developed over the course of the term. This helped to facilitate the academic knowledge acquired from the User-Centered Design course (SYDE 348) and offered first-hand experience of the advantages and disadvantages associated with several usability techniques. Furthermore, the UW library gateway project promoted a better understanding of how to approach a usability analysis of a web site, which will definitely become useful in my upcoming work term.
A team approach to the UWL gateway project was beneficial because it provided an opportunity to develop and further refine fundamental team management skills involving leadership, decision making, project planning, and establishing group norms. Furthermore, working in a team promoted a better understanding of the necessary requirements for achieving a co-operative learning group as well as the requirements for achieving a high-performance co-operative learning group such as the need for equal participation, respect for other’s ideas, careful listening, and a high level of commitment to the project as well as to group members.
Veronica
This project allowed for a better understanding of the difficulties of working in a large multi-disciplinary team. One of the major difficulties involved scheduling because many of the group members had different schedules, meeting times where all members could attend, were difficult to arrange. As a result of this, there was and increased need to have things done on time. Team members totally depended on each other, and if one persons work was not complete, the preparation of a report would be postponed. In response to this, the benefits of setting early deadlines was learned because no matter what, Murphy’s Law comes to play and you end up rushing to the end no matter what, it seems.
Having group members from different academic backgrounds allowed for different views of the design process. I became more aware of the Systems Design program and their academic styles. I enjoyed the different views and ideas that resulted from their different backgrounds.
I also learned the importance of group meeting for conducting open dialogue, and organizing group activities. Another means of organization that I learned about was email. It made communication easy, and kept inbox full.
Another important lesson was compromise. It is really important to quickly learn that you can’t always get what you want. Sometimes you have to give a little to get a little. In addition, time also had to be compromised due to the differing schedules amongst the group members.
Finally I learned that if you don’t understand something, ask questions right away, otherwise you will be lost unnecessarily, and this will make task completion difficult if not impossible.
Barry
The interdisciplinary composition of the team allowed us to complete a project that crossed a wide boundary of tasks. For example, the help of recent ergonomics students allowed others to refresh their minds about ergonomic problems, and such issues were dealt with in quick order. Without this infusion, the team might have overlooked some ergonomic factors altogether.
The number of people on the team allowed us to more evenly divide tasks and alleviate individual burden. With smaller groups, an uneven workload can affect individual performance and consequently, group dynamics.
Deadlines were important, especially when the team could not meet often enough due to geographical and scheduling reasons. The interdependencies of group members require that portions of work be completed on schedule.
Pam
Individual learning accomplishments: The most significant learning objective accomplished in this project was the formal execution of various usability testing methods. A total of four user-testing methods were used throughout the project to determine user needs and to test various stages of our prototype. I was involved in the execution of three of the four user-testing events: Design Walkthrough (user requirements gathering), Card-sorting technique (user requirements gathering) and Keystroke Analysis (evaluation of functional prototype).
The usefulness of each testing event was confirmed by the fact that the results from each event revealed different interface design issues. It was also very insightful to see how some of the results obtained in one testing event contradicted those obtained in a different event. For example, the content organization derived from the card-sorting tests appeared somewhat unintuitive in the keystroke analysis test. In addition to this, while users in the discount usability tests preferred to ‘click’ each main option button in order to view the corresponding list of contents, several users in the keystroke analysis test suggested that the content list be displayed on ‘rollover’.
The discrepancies between usability test results emphasize the need to use several methods when testing the user-interface in order to ensure that the proposed solution is evaluated from ‘all angles’.
I am also very pleased with the opportunity of implementing the functional prototype. The implementation process involved designing the aesthetic (graphical) component of the interface and coding the functional features using HTML and java-script
Team-related learning accomplishments: In order to complete all project tasks and milestones in a timely fashion, the leveraged on each individual member’s strengths and abilities when allocating several project tasks. It would have been ideal to pair an ‘expert’ member with another member who was interested in learning about a particular area. However, this ‘transfer of abilities’ was often not possible due to the tight project deadlines and high workload volumes of the individual members.
Another aspect of teamwork that I learned in this experience was the importance of finding the most appropriate time to voice my opinions and concerns. Since our group was composed of very outspoken and less outspoken individuals, it was imperative that the all members took time to listen to the views of the less outspoken members. This practice prevented ‘individual dominance’ of the more outspoken members.
Simon
I whole-heartedly feel that I have learned a great deal about patience and tolerance. I strongly believe that there were many situations I encountered while working in the group that would have in the past caused me either to lose my temper or resolve in a unprofessional manner. More specifically I learned how to deal with conflict with certain members of the group. I feel that the working with such individuals though not enjoyable, allowed me to grow as a person and learn how to solve conflicts. Thus I feel that conflict resolution is the greatest learning objective I have met.
Secondly, working as a team has allowed me to participate as a team player as opposed to a leader. Most of my experiences with teams in the past have involved self leadership as opposed to actual teamwork. Though things were done the way I wanted them to be, I didn’t learn to open my mind to new ideas and methods of accomplishing our goals. I have learned the valuable skill of being a team player as opposed to being the sole leader.
The team showed me how a small group of people each with their respective strengths can come together to meet a common goal. It further emphasized the power of teamwork and dedication and how these are powerful tools to accomplish any task.
Finally, I feel that all members of the team benefited from the realization that each individual has their own skill sets and own ways of accomplishing a task. Myself, I found out that there are many ways to do something and that no one way is the only way. Furthermore, I found out later in the project that I must rely on others to complete their part and not to do everything myself.
I think that if we as a group were assigned to do another project that there would be one main thing I would do differently. That is to establish the roles of each team member as being equal. We should all be given a fair chance to speak our mind and to offer opinions which was not the case for this project. I would like to have made sure that a rule was set to allow all members feel free to give an opinion or speak freely without being interrupted or scrutinized by other team members.
Julia
The most important thing I learned from this project was time commitment. Since everybody had busy schedules and we all had to compromise to find times to meet. Communication was a huge issue. Email was good, but I wondered if everybody reads it all and sometimes I wondered if it would have be better to talk in person.
The project itself let me see how much work, consideration, and testing goes into user centre design projects. I liked the idea of logbooks because it was a way to show the progression of what happened and helped me keep up with what was going on with the project. In the future though, I think I would include more in the book because I had never kept a design book before this, I didn’t know what to write in it.
This project also gave a practical side to the theoretical components of the course. The teams allowed a lot of work to get done. I learned that when working with a team, trust is a big issue. You have to be able to rely on your team mates to get the work done that they are assigned. To help with this, having group norms were necessary so that people didn’t do work half-assed!
Another thing I learned was that everybody has different ways of doing things, different skills, and you have to capitalize on these differences to make things work. It is important to remember that you can’t focus on the bad things of people’s differences.
In the future I think I would try to trust people more. Sometimes I wondered if I was individually dominating in the group.