Below is a list of activities the targeted users would perform on the Library Gateway page:

• Look up information on library materials
• Search and retrieve journals
• Browse links for external sites
• Reserve books from various libraries
• Renew/request books
• Course reserve lookup
• Search catalogue
• Use reference tools
• Read library news
• Find library information (e.g. hours of operation, phone extensions)
• Provide feedback to the library
• Find information on services available

All of activities would be used and should be supported. The frequency of each activity should be taken into account to determine the importance of that activity to the web site. This will help understand where the focus for improvements should be.

The most frequently performed activity should have direct links on the Library Gateway web site. For activities that are not as frequently performed, they should be categorized into groups and should be available as a secondary links.

This section categorizes the user base of the Library web site:

• Undergraduate Students - Low familiarity with using existing library resources.
• Graduate Students - High usage of library resources.
• Faculty - High usage of library resources.
• Staff - Low usage of library resources.
• Librarians - Expert knowledge of the library.
• Maintenance and Support Staff - Primarily administrative uses of library web site.
• Non-UW users (local community) - Access of library resources from off-campus.

• Novice library gateway users
• Expert library gateway users
• Experienced computer user
• Advanced computer user

The ratio of the different user groups should also be considered to understand the proportion of users of the library resources to look at meeting the needs of the current users or improving usage for a smaller group.

General Requirement

The final design should suit a maximum number of users by looking at their common tasks and improving them.

The following section describes the extent to which the proposed solution would support:

• Easy to navigate
• Appropriate level of language usage for both novice and expert users
• Consistent layout
• Help on various topics
• Customized search within the library site
• User friendly

The support requires input from the library staff as a whole to deliver information to the users.

This section proposes the form of the final outcome of the design activity:

• University of Waterloo Library Web Gateway page
• Template for related library web pages
• Compatible to the most common versions of browsers
• Compatible to most common resolution settings

Due to time constraints, only the proposed design of the UW Library Web Gateway should be implemented, while recommendations on the general Library related web site should be made without implementation.

This section describes all the assumptions and constraints of this project. Although these assumptions may seem to restrict the range of possible solutions, they help to maintain a manageable project scope.

1. Teams were given one academic term to apply the spiral method technique to test a redesigned library gateway page. This time constraint played a significant role in term of how much technical implementation detail the prototype would include. However, it was assumed that a functional prototype would be developed, and tested with users.
2. It was assumed that the performance of the web site, in terms of load time, was beyond the scope of this project.
3. It was assumed that the main gateway and subsequent layers would be evaluated when completing various usability methods. However, only the main gateway was to be redesigned. Therefore, recommendations on the sub-layers would only be made with respect of overall consistency.
4. Due to budget constraints, all implementation was done in-house (i.e. within our group).
5. Budget was not allocated for purchasing of new web authoring tools. Hence, implementation was limited to the tools available within our group.
6. Only two out of six members on our team had experience with web page design. However, some technical support was made available from our TA and Professor.
7. It was assumed that none of the team members had any previous exposure to User-Centred Design (UCD). UCD knowledge that was implemented within the design of the gateway was gained during the term.
8. It was assumed that the course instructor would outline most evaluation methods performed on the prototypes, with some variability in phase two.

1. The proposed UW Library gateway design should support, as a minimum, the functions provided on the current web site.
2. The proposed UW Library gateway should support the following target user groups with novice or expert library gateway experiences as well as experienced and advanced computer users:
   • Undergraduate students
   • Graduate students
   • Faculty
   • Staff
   • Librarians
   • Maintenance and support staff
   • Non-UW users (local community)
3. The final design should be compatible to the most common versions of browsers.
4. The final design should be compatible to the most common resolution settings.
5. The final design should be easy to navigate
6. The final design should be user-friendly.
7. The final design should use appropriate level of language to suit both novice and expert users.
8. The final design should demonstrate consistent layout with the entire UW Library site.
9. The final design should provide help on various topics.
10. The final design should provide customized search within the library site.
11. The final prototype should be functional on the main gateway level including click-able links.

Heuristic evaluation was chosen as one of the primary methods of evaluation in Phase 1. Heuristics are the "rules of thumb" which aids the designer in identifying usability problems. This in turn facilitates the correction of these problems early on in the iterative re-design process. In addition, because these heuristics are general in nature, they are effective in identifying problems rather than instructing the designer on how to solve them. This in turn allows for more creative problem solving and generation of solutions. In other words, they do not limit the final solution by specifying a correction which may "pigeon hole" the evaluators, preventing them from coming up with additional ideas.

The heuristic evaluation completed on the gateway involved having two evaluators examine the interface and judge how well it complied with Nielsen's ten recommended usability heuristics (Nielsen, 2001). Refer to Appendix 1 for detailed listing of the ten heuristics. Due to resource constraints, two evaluators were used in the inspection process in order to ensure more thorough and extensive identification of usability problems though it would be ideal to have at least three evaluators since different evaluators tend to find different problems (Nielsen, 2001). Nielsen's ten recommended heuristics (Nielsen, 2001) were selected over other methods for the preliminary evaluation of the UW Library gateway for several reasons. Firstly, these heuristics outline the most important aspects of usability as specified by a well-known usability expert with extensive experience in the field. Therefore, they present a high degree of face-validity. In addition, Nielsen's heuristics (Nielsen, 2001) cover a very broad range of usability issues, thereby ensuring the identification and correction of as many different types of problems as possible.

The two evaluators individually evaluated the interface by carrying out the inspection in a systematic manner. The usability experts began with the first of the recommended heuristics and inspected the gateway for any aspects of the interface that did not conform to the "rules of thumb" specified. The inspection began with the first mouse-over ("Find It"), where the evaluators moved through the available options. Following this systematic navigation of the system, the evaluators randomly moved through the gateway and its associated pages to identify any additional problems missed. Each of the evaluators repeated this process for each of the remaining nine heuristics. The evaluators recorded usability problems during the inspection. It must be noted that the evaluators did not converse or share their finding in any way until the inspections were completed. This was done to ensure that the evaluations were completely independent free from bias. The heuristic evaluation session was completed in approximately three hours.

It is important to note that upon identification of any usability problems, the evaluators stated why it was a problem with specific reference to the usability heuristic that was violated. Specification of why a particular aspect of the design was a usability problem was critical because it would allow for easy revision of the current design during the iterative re-design process.

Cognitive Walkthrough

Cognitive walkthrough is a usability method performed by expert evaluators to inspect the interface. In this method, the expert thinks through the steps that the user would most likely do when performing a task. This can give the expert a sense of the extent that the interface is user friendly. If the expert finds it difficult to complete a specific task, it may be an indication that there is a problem that needs to be fixed before more expensive and formal testing performed. Cognitive walkthroughs are used because they are informal, fairly quick and are capable of flagging potential problems.

One expert evaluator completed the cognitive walkthrough. For this particular design project, the clients provided a list of Web Usability Questions that they thought were common library user scenarios. The evaluator tried to answer the twenty questions supplied by the client in sequential order. Since one of the suggestions from the client was to reduce the number of click in a specific task, it was felt that in order to accomplish this, accounting for the current number of clicks to answer each question would be helpful. To measure the success of the gateway in terms of this goal, the same questions can be posed to users once the new design of the gateway is complete the task was also recorded.

Once the evaluator read a question, the start time was recorded. The evaluator then tried to place themselves in the user's shoes and performed the steps they thought a typical user would likely follow in attempting to answer the question. Each click required to manoeuvre through the pages was manually recorded in a tally chart until the task was completed. Upon completion, the clock time was stopped and the total time taken to complete the task was recorded. Comments on the ease or difficulty of the process were recorded following each question. If a specific task was taking a long time to complete and the evaluator did not feel that they would be able to find the answer, the task was halted. The time, errors and difficulties faced were all recorded for further analysis. This procedure was repeated for each of the other questions.

Task Analysis

The final inspection method used was the Hierarchical Task Analysis (HTA). An HTA is a decomposition of a task into its component building blocks. The goals, plans, and operations of a user are recorded into a hierarchical structure. This provides a task description similar to what may be found in the cognitive walkthrough. The analysis involves the identification of the steps where there may be usability problems.

A task analysis session was used to analyze the performance of tasks over the entire site as a whole. The Web Usability Questions that the clients specified were further refined in order of importance. The top five questions were redefined as specific tasks in the use of the library web site.

A breakdown of each task was done to record the general plans of the tasks under investigation, see Appendix J. After each plan was completed, the analysis phase with a second team was conducted to validate the task flow. This HTA outlined the task flow of the current UW Library web site to identify which tasks would be inefficient for a user to conduct.

The five tasks that were analyzed were:

1. Find the book Alias Grace, by Margaret Atwood at the UW Library.
2. Find out if the University Map and Design Library is open on the weekends.
3. Find the librarian for Systems Design Engineering.
4. Find a database in which to locate articles on Anthropology subjects.
5. Get a copy of Stanton from a remote library.

Competitive Analyses

The intent of this method was to compare different gateways of different universities and to conduct a competitive analysis of each of the web pages. The competitive analysis was composed of the important aspects that would be necessary on the re-design. They consisted of overall site layout, graphics, mouse over versus text layout, dead spaces, labels, navigation, layout consistency, and clarity of top navigation bar. Our team carried out the method by assigning one of the team members the responsibility of analyzing and recording the design of each gateway. The member first examined the web site gateway and then addressed each of the areas mentioned above. Each of the questions were answered according to the team member's point of view and their results were then discussed as a group.

In Phase 1, heuristic evaluation, cognitive walkthrough, tasks analyses as well as the competitive analysis were employed to determine the usability of the UW Library Gateway page. These methods used different techniques for evaluation but revealed similar insights about probably user interactions with the web site.

Below is a summary of insights gained using the three different methods. The problems discovered are categorized into Nielsen's Ten Recommended Heuristics (Nielsen, 2001).

Visibility of System Status

This heuristic evaluation discovered the problem of lack of visibility of the Library Gateway. Certain links on the Gateway page link to external sites. These links give no indication to the users that he/she will be leaving the UW Library site. As well, the UW pages do not consistently identify themselves as an UW page, making it difficult for the user to know when they have left the UW web site.

There is a discrepancy in the terminology between the Gateway system and the physical Library. The concept of "Find It" and "Get It" have proven to be confusing in requiring lots of time for remembering and analysing what the correct heading for a particular search may be. Both task analysis and cognitive walkthrough support the errors found through heuristic evaluations. "Find It" and "Get It" do not match the concept of physical searches in a brick and mortar library.

Navigation controls are sorely lacking this version of the UW Library Gateway. There are no methods available to move between different web pages along your chosen path. A mechanism to return to a particular type of search, like "staff search" from the found results would be helpful.

The UW Library site has poor consistency and standards across the site. Fonts and colours are inconsistent; the user often leaves the UW site without knowing about it. Also there are areas where the same page may be reached through two different links with different wording. An example is About the Library: Accessibility and Services for: Persons with Disability, leading to the Services for Persons with Disabilities page.

There is nothing that helps the user in reducing the number of errors made. The links are poorly worded which forces the user to conduct trial and error until they learn the system. Any task that did not involve obvious "Find It" required a scan through all of the mouse over menus for the user to guess at the proper link.

With the mouse over style menu, the items in the menu disappear as the next item in the list is selected. This creates a burden on the user's memory and with the difficulty seen in Error prevention; the site further frustrates the user by demanding too much memory.

There is no support in this version of the web site for efficiency. An example is the advanced search. The site does not provide additional fields to search. The person conducting the search much input his or her own special codes for an advanced search.

From the heuristic evaluation, the picture of the Davis Centre (DC) Library on the Gateway is distracting from the functionality of the site. As well, too much text is displayed in some of the pages.

In this site, there are very few methods for allowing the user to recognise error. The site mostly depends on the user interpreting their results from the links that they visit or their searches to realise that they have made an error. The only recovery so far is the browser's back button.

The "Ask Us/Tell Us" button at the top navigation bar is quite visible and accessible to users. However, there is no Frequently Asked Questions (FAQ) available for a general help.

Solutions targeting the problems discovered in the previous section were proposed. Alternative designs were implemented based on these suggestions.

Figure 1: Low Fidelity Prototype 1

Figure 2: Low Fidelity Prototype 2

Graphics on the Gateway

There were many complaints that the three dimensional picture of the DC Library was distracting when trying to read the labels. Another problem with the graphics is that they are asymmetrical. It only covers the left side of the page leaving the right side with a lot of white space. Also, the graphics on the main page hide the link to the text version of the web site. The arrow that appears when the mouse if hovered over any one of the links is not a distinctly identified as an arrow. Finally, there is too much space between the fixed links on the left side of the page and the box that appears when the mouse is hovered over any one of the fixed links. It can be confusing to the user if it is their first time using the gateway. It has also been decided not to use a distracting image in the background, instead a black background was chosen so that it would contrast well with the yellow/orange fonts used. In addition, the link to the text version of the web site should be clearly be seen as it is displayed in a white font against a black background.

From the primary usability evaluations carried out in Phase 1, the mouse over layout does not appear to have any advantages over the text layout. With the mouse over layout, the user is forced to remember the listing in each heading because it disappears as soon as the mouse is moved away from that heading. This makes it difficult for the user to decide which pathway to follow since there is only one set of options displayed on the screen at any given time. The move over version also has many limitations when being viewed at different resolutions. It seems to be optimal in terms of layout and spacing when being viewed at a resolution of 800 by 600. At any resolution lower than 800 by 600 it is essentially impossible to view the 'pop-up' listing while hovering over one of the main headings. At any resolution higher than 800 by 600, the screen is not used efficiently (i.e. there is too much white space). The text version adjusts itself so that it maximizes the use of space on the screen at any resolution.

The main distinction of the alternative design is that the use of mouse over will be eliminated. A comparison of the text layout and the mouse over layout illustrated some of the design flaws in the mouse over layout. In getting rid of the mouse over layout, the limitations of viewing the page at only certain resolutions will be eliminated. As illustrated in the two prototypes created, the web page consists of only links (refer to Figures 1 and 2). All of the possible pathways (links) are always displayed on the screen so the user can quickly scan through the headings and decide which ink is needed to perform their desired task.

The entire right side of the page is wasted in the mouse over layout. The higher the resolution in which the page is viewed, the more white space there is on the right side and at the bottom of the page.

White space should be minimized by the use of the graphics. In addition, dark background colour with a light coloured text should be used to enhance the contrasts, readability, as well as attractiveness.

As depicted in the two prototypes, both of their backgrounds have been chosen to be dark colours with a lighter coloured text. It was believed that changing the background of the page would make it more attractive to the users. The layout of the pages are also re-designed to maximize the use of space and eliminate any dead space. As illustrated in Figure 1, the graphic of the Waterloo warrior is included.

The language used for the labels is not very intuitive to average users. By reading the label, the user should have a general idea of what can be found under that heading. This is not always obvious with the current labels. It would help if the titles were more descriptive rather than being so general.

The labels in the next prototype should be more intuitive to present the user with a better understanding of what is behind that label. Since it will be a text-based layout, the user will be able to see all the options available to him/her, similar to the text version of the current library gateway. As illustrated in Figures 1 and 2, both prototypes consider these recommendations and hence use more intuitive wordings to eliminate the discrepancies encountered by the users when searching for the appropriate links.

There is very little consistency among the pages in the UW Library web site. Most of the pages include the top navigation bar but not all of them do. For example, if you go under "Find It", Journal Indexes and then click on one of the links under the Broad Subject Areas listing, the top navigation bar disappears in the new page. Instead, there is a TUG Libraries home page (www.tug-libraries.on.ca). From here, there is no link that will take you back to the main UW library gateway. This can be very confusing for even an advanced user, let alone a novice user. Given the lack of consistency, it is difficult at time to tell whether the user is still one of the UW Library pages.

Another navigation problem is that when the user goes to an external site, it does not open up in a new window. A novice user might get confused as to how to get back to the gateway page without having to re-type it in the address bar or without using the back button on the browser. A solution to this problem would be to implement something similar to Microsoft's Hotmail site. When you click on a link within Hotmail, a new window opens up and the following phrase can be found clearly at the top of the window "You are visiting a site outside of Hotmail. Close this new browser window to return to Hotmail."

In addition, there is no mechanism for tracing the path taken from the gateway. This makes it very difficult for users to get to intermediary pages if they run into a dead end. Currently, the only way for one to get to an intermediary page is to use the back button on the web browser or to remember the steps taken from the gateway.

In the next prototype, the use of a quick search was always available to the user so that advanced users can find what they're looking for as fast as possible. As well, since the sub-pages were not within the scope of this project, they were not developed.

Essentially, the layout in the next prototype was to be text-based with the use of a navigation frame on the left side of the page as designed in our prototypes (see Figures 1 and 2). This navigation frame would allow the user to know their position in the web site and be able to navigate to higher or lower levels from their current position. This also eliminated the need for the user to remember the path that was taken from the home gateway.

The top navigation bar hardly helps the user 'navigate' through the site. It allows you to get to the gateway page at any time and it allows you to view a site map or to search the UW Library site. This search tool is not a very effective one. When searching for the word TRELLIS, 2949 documents were found. Instead of displaying the main link to the TRELLIS page in the search result (which is what most people would be looking for when performing such a search), it displayed every page that had TRELLIS in the URL.

In the proposed prototype, a brand new top navigation bar will be created to aid users in navigating the UW Library related sites. Specific headings include Home, TRELLIS, Quick Links, and How do I.? This can be seen in Figures 1 and 2.

Intent of Method

According to previous testing results, we found that the language used for the labels were not very intuitive to average users and we indicated that when reading the labels, the user should have a general idea of what can be found under that heading. This was not always obvious with the current labels on the library gateway. It would help if the titles were more descriptive rather than being so general.

Card sorting is a common usability technique that is often used to discover users' mental models of an information space (Nielson, 2001). Links that get sorted together should be grouped into the same menu. A downside of the method is that users may not always have optimal models, and card sorting is often used to assess the difference between the way novice and expert users understand a system (Nielson 2001). The names of category that the test participants came up with provided some insights into their mental models regarding information layout and allowed us to better categorize each of the links.

Method

Card sorting was performed by giving each of the users pieces of paper with concepts and links from the Waterloo Library home page and secondary links written on each one. Each individual was asked to sort the cards into piles; the links representing similar concepts were placed in the same pile. Each concept/link were keywords taken from the current library web page. There were six users that performed this card sort; three undergraduate engineering students and three undergraduate kinesiology students. The group members were used as the users and the task was performed individually. The users was instructed to sort the concepts/links into two piles and asked to indicate the reasons for sorting them in this way. Users then were told to sort them into four piles, then six and lastly into whatever number of piles they thought were appropriate; again indicating the reason for sorting them in that manner. Once all the users performed the card-sorting task, the results were analyzed and discussed as a group and the group performed a final card sorting. This card sorting took into account everyone's ideas and suggestions and the final result were used for the medium fidelity prototype. Only group members were tested due to the fact that no ethic approval was sought for this particular activity.

After performing the card sorting technique on each of the group members, we found the following characteristics. When two categories were used, it was generally found that all the search tools were combined into one category while the other category consisted of basically user and library information/links. It was found that two categories were not sufficient enough to allow the user to navigate efficiently in the gateway.

When the users were asked to divide the links into four categories, it was found that in most cases there was a 'users' category, a category to search information/materials, an about the library category and other links category. Sorting the links into four categories was found to be difficult at times since not all the links necessarily fit into any one of the four categories. It was generally found that users required more categories to sort the links into.

When the users were asked to divide the links into six categories, it was discovered that the users found it much easier to place links into corresponding categories. They had headings that related more to the link allowing one to navigate to the appropriate category. The common headings created in this task were as follows: a user category, an other libraries category, a searching for resources/finding information category and about the library category. The last two categories varied from help/how to, secondary services, tools/ tools for you and other links.

When the users were asked to complete a card sort using any number of categories they felt to be appropriate, many different results were obtained. Two users chose five categories, one chose six categories, one chose seven categories and one chose eight categories. Everyone had different ideas as to how many categories would be optimal.

After each user completed their card sorting individually, we sat down as a group and worked together to come up with what they thought would be optimal. We decided upon a solution using six categories.

Often times, usability experts will insist on using the best possible methodology; this however, may not be realistic depending on the resources they have available to them. Time constraints and financial limitations can very easily preclude the use of the most thorough, accurate and extensive methods. When this is the case, discount usability can be utilized. Discount usability considers both financial limitations and time constraints and works around them; it is an evaluation method which combines heuristic evaluation of the proposed product, and scenario testing, where informal design walkthroughs or verbal protocols can be utilized (MacGregor, 2001). This particular method provides an adequate means of usability testing and can provide fairly good usability data due to the fact that it forces the testers to fix problems and retest to see whether the changes are appropriate or not. They allow for quick/immediate feedback from users on the problems and for fast re-design and testing. It is true that better results can be achieved by applying more careful methodologies, but the use of more meticulous methods would cost more expense in terms of money, time, and required expertise. Therefore, the simpler methods outlined in discount usability stand a better chance of being used in practical design situations (MacGregor, 2001).

Heuristic Evaluation

Intent of Method

Nielsen's (2001) ten recommended heuristics were selected over other methods for evaluation of the low-fidelity prototypes. Our group had benefited from using the heuristic analysis in the previous stage and decided on its use again to identify possible interaction problems with the prototype gateway.

Method

The heuristic evaluation completed on the two low-fidelity prototypes involved having four evaluators (from the design team) to examine the interface and to judge how well it complied with Nielsen's ten recommended usability heuristics (Nielson, 2001). The four evaluators individually evaluated the interface carrying out the inspection in a systematic manner. The usability experts began with the first of the recommended heuristics and inspected the prototypes for any aspects of the interface that did not conform to the "rule of thumb" specified. The inspection began with the first mouse-over "Your Research Needs", where the evaluators moved through the available options. Following this systematic navigation of the system, the evaluators randomly moved through the prototypes to identify any additional problems missed. Each of the evaluators repeated this process for each of the remaining nine heuristics. The evaluators recorded usability problems during the inspection. It must be noted that the evaluators did not converse or share their findings in any way until the inspections were completed. This was done to ensure that the evaluations were completely independent and free from bias. The heuristic evaluation session was completed in approximately three hours.

It is important to note that upon identification of any usability problems, the evaluators stated why it was a problem with specific reference to the usability heuristic that was violated. Specification of why a particular aspect of the design was a usability problem was critical because it allowed for easy revision of the current design during the iterative re-design process.

Design Walkthrough

Intent of Method

The second part of the Discount usability is the Design Walkthrough. Completing a Design Walkthrough by actual users allows potential problems with the present prototype to be identified. It is very important for the designers to gather insight into the steps taken by the user to solve a specific problem or find the answers to a question. It is very difficult for a designer who is essentially an expert user to place themselves in the mind frame of a novice user. They are unable to perceive all the potential problems that an actual user may encounter with the prototype (MacGregor, 2001).

The Design Walkthrough allows an evaluation of whether the correct actions are sufficiently evident to the user. The sequential steps required to complete a task, should be visible to the user, which should be addressed in the layout of the design. Another question design walkthroughs address is whether the user is able to connect the correct action's description (e.g. label, dialogue boxes etc) with what their task is? The results may indicate to the designer that the labels need to be more descriptive so that the choices become more intuitive. Lastly, Design Walkthroughs will let the designers know whether the user interprets the system's response correctly. Through these findings, modifications may be found necessary and users can give their feedback as to improved feedback from the system (MacGregor, 2001).

Method

Design walkthrough was performed as part of the discount usability testing on the two low fidelity prototypes. The two low fidelity prototypes tested were implemented using the PowerPoint. Five users in total participated on this testing. These participants were selected on a random basis from computer labs in two faculties, engineering and applied health sciences, in order to have different user levels. These users represented the population of the undergraduate student body from the Engineering, Math, Science, and the Arts Faculties. Most of the subjects tested were novice users of the library site. Further testing was planned for faculty and expert users for the next phase.

At the beginning of the testing, the user was asked to answer six questions that were chosen based on the recommendations of our client. These questions addressed tasks commonly performed on the library gateway. The questions that were used for the design can be found in Phase 1.

Each user was asked to answer these questions one by one and to point out the link on the prototype which they would go to in order to find the information the question was asking for. The users were encouraged to think out loud of what was going through their minds as they were looking for the answer. The user was asked to try a different link if their choice was different from the one the design was intended for.

Second Design Walkthrough on the Modified Prototype

Method Details

Based on the previous findings, we combined the two prototypes into one by incorporating the proposed changes raised in the last section. Design walkthrough was then performed on this refined prototype. Five different users participated in this walkthrough. The procedure for this testing was carried out the same way as the walkthrough on the last prototype. However, different questions were asked and they targeted the areas we have improved on this prototype.

After novice users completed the walkthroughs for the two prototypes, it was apparent that changes would need to be made to our medium fidelity version. One positive aspect to finding problems with our low fidelity prototype is that many of the users found the same problems. These problems are addressed more specifically.

Prototype #1 (Black Background)

A consistent problem among users pertained to their focusing mainly on the left margin, ignoring the main contents to the right when attempting to answer the specific questions during the walkthrough. This could be expected due to the fact that participants are used to reading from left to right. Once they start focusing and reading the contents in the left, they forget that there is more information in the gateway. In order to address this problem the design team decided to move the left margin and its contents to the right, leaving the user to focus initially on the intended main contents of the homepage.

Another problem with the left margin was that the specific headings were similar, but not identical to the contents on the right, producing some redundancy and confusion. Specifically, "Your Research Needs", "Services For You" and "Tools for You" appeared in both sections of the page. The logic behind the original design was that the contents on the left would reflect what we thought was used most often. Following user testing however, it was found that the headings and links should appear only once on the Gateway page. The proposal for change was to place the central links in the main part of the page (which was just moved to the left), and only the most important links would be placed in the right margin. Essentially, the right margin is focused for the expert user.

Comments were made about the excessive information overload on this page as opposed to Prototype #2. This was addressed by decreasing the number of links in the margin. We limited the links to the items used most often. There was also a general problem with the font size in the left margin being too small. This made it difficult for the users to read the links effectively and efficiently. By reducing some of the contents in the margin area, the font size can be increased. Aesthetically, the margin will look less cluttered.

It was noticed that there were problems with the specific wording of the main headings. The heading "Library Links" was confusing because users could not decipher whether it was information specific to our library or other remote libraries. The design team decided that a more descriptive heading would be "Links to Other Libraries". Instead of titling "About the Library", we decided to make it more specific so that there could not be any confusion as to what is found under this category. Thus, we decided to call it "About the UW Libraries". The heading "Quick Link" was confusing for some users because as they pointed out, the links in the left margin were essentially quick links themselves. Therefore, what was the purpose of this button or contents found there? Since there was so much confusion with this button, it was taken out altogether in the next prototype.

It was observed that people got confused between the "Services for You" category and "About the Library". This was apparent when users were asked to look for library times/location. Instead of going to "About Library", they looked under "Services". Thus, we are thinking of placing the "Services" under "About Library". The links under "Services" are not things that are used on a regular basis. Realistically, participants may look at it once, as a new user to find out what is offered to them.

For the quick search, individuals preferred the availability to search by a specific medium (e.g. journal, books, e-journals...), similar to what was available on prototype #2. Having the drop down box with pre-set categories following a topic that was typed in by the user seemed to work well for the users. To continue with trying to be more descriptive, a title that reads "Library Forms" should be placed on top of the TUGdoc drop down box, like the original UW gateway.

Prototype #2 (Blue Background)

A major complaint was that the information under "What's New" was not apparent to the user and did not need to be there. It seems that currently, the Library does not do a good job of keeping this information updated. The specific questions that we asked in the walkthrough did not require the use of any of the links presented under this category. All the questions seemed to pertain to the links on the left margin. It was apparent to the designers that information in the main panel needed to be changed. We decided that "What's New" at the library was not important enough to place in the gateway.

The heading "Quick Links" category was not very informative. The users did not understand why the links were exclusive to Laurier and Guelph Libraries when all the links on the left margin were quick links as well. As suggested earlier, this link was deleted altogether. The users will still be able to access Laurier or Guelph from "Links to Other Libraries".

It was observed that there was no "How Do I...?" or help function like there was in prototype #1. Thus, in the new revision, a Help function will be included. Following the link, "Our Catalogue" the word "Trellis" should be included in brackets to support users of all experience. More advanced users know to look for "Trellis" while more novice users may not know the name of our catalogue, but understand that this is where they need to go to search for specific items by subject, author etc.

Overall, the participants found that aesthetically, prototype #1 was better. However, they preferred the simplicity and functionality, of prototype #2. From these findings, it was determined that the next prototype should incorporate the advantages of both prototypes #1 and #2.

It was found from the design walkthrough testing that the top navigation bar was not used very frequently. Some users did not even realize the existence of this top navigation bar while some users were not sure if those were icons that they could click on.

Some users commented that it would be more efficient if the Warrior bubble would display the actual underlying links rather than just plain text description of what was underneath. This would save the number of clicks for users.

The "Contact Us" button on the top navigation bar was not visible to most users. Most users expected this link to appear on the right hand column as a quick link. We added the "Contact Us" link to the quick links column on the right hand side for our next medium fidelity prototype.

It was confirmed that the "By subject/ book" link on the right quick links column aided the novice users in looking up books as they usually do not understand the meaning of "catalogue" nor "Trellis". The "Trellis - our catalogue" link was still kept for expert users who are familiar with these terms.

The "How do I..." link was proved to be useful in instances when the users were not sure where to go to.

The "Your Research Needs" link was commented as ambiguous and did not imply any direct meaning to the users (both expert and novice users). We would rename this link and call it "Find book, journals, data..." instead.

The meaning of "Services for you" was too broad for the users. Some users perceived that this link would include information for all the services that the library provides, such as renewals, getting books from other libraries, etc. Rather, the "services for you" was intended to link to information on services provided for various group of users (e.g. undergraduates, graduates, etc.). We decided to take out this link from the right hand column since they are not accessed frequently enough to be put on the quick links column.

Based on the findings of the second design walkthrough, changes proposed in the previous section were incorporated in the medium fidelity prototype. This prototype was tested in the next project phase.

Figure 3: Proposed Medium Fidelity Prototype

Lab-based usability studies involves testing the usability of a web site in a laboratory situation (Internet Resource Centre, 2001). A selection of users are recruited and placed in front of a computer where they have access to a web site. These users are asked to undertake specific tasks on the site where their reactions to the web are recorded and analysed. In addition, participants are interviewed about the web site and their feedback compared with the analysis of their recorded site interactions.

Conducting lab based usability testing provides a means for the design team to investigate issues in the future product. Often times, the designers are not quite sure of the interaction between a web site and its users and these specific elements can be tested. For example, running two sets of controlled tests on mouse over help text allows the designers to see the performance increase or decrease due to the addition or elimination of help text.

This method is designed to gather information from actual users rather than relying on a heuristic analysis by an expert. It is better to have the actual user working with the web site rather than a reviewer predicting how users will respond. This advantage was not apparent during the early iterations of the UW Library Gateway, since the expert conducting the heuristic analysis was also a designer of the web page and may have been biased towards the page. However, once ethics approval was received, testing on actual users were done and problems or insights beyond the designers' scope were brought forth to aid in improvements for future prototypes.

The specific method exercised in lab based usability testing is outlined in the ORE forms which is included in Appendix A.

Participants:

A total of 10 undergraduate students from the University of Waterloo were tested, 6 of which were females and 4 of which were males. The participants that were tested came from varying faculties including Systems Design, Arts, Biology, Math, Kinesiology, and Computer Science. The age range of these participants was from 20-24, with an average age of 21.9.

The averages score across all questions in the Background Questionnaire for each individual participant were calculated to determine their level of expertise of the overall Library Gateway. Please refer to Appendix A for the specific questions asked. Users were operationally defined as follows:

Novice 1.0 - 2.0

Experienced 2.1 - 3.9

Expert 4.0 - 5.0

It was found that the majority of the users (7/10) were experienced users whereas the remaining participants (3/10) were expert users.

Procedures:

As a group it was decided that an equal number of team members would utilize data collection task list version A and B. The questions were asked in an upbeat tone, with no emphasis placed on specific words. This was to ensure that participants were not in anyway influenced in their answers to the questions. The questions were also read in a clear deliberate manner to guarantee that questions were heard and understood.

The participants were recruited prior to the predetermined date of testing. Once the participants arrived in the computer lab, they were greeted and brought to a computer terminal. The library gateway page that was to be tested was minimized so that the participants did not have the advantage of seeing the prototype before the actual testing began. All users were asked to read and sign the Information/Consent Letter prior to the start of testing. See Appendix B.

All participants were then asked to fill out a Background Questionnaire concerning their familiarity with the UW library gateway page and web pages in general (See Appendix C). Test monitors then read from a set script to ensure that each participant received the same set of instructions. Users were reminded that the objective of the study was to test the usability of the design and not to test their personal skills (See Appendix C). The participants were asked to perform a set of tasks. More specifically, they were asked to answer 10 questions related to common library activities using the redesigned gateway page.

Test monitors were required to time how long it took the users to complete the tasks. If mistakes were made during the execution of the task, the monitors were to pause the stopwatch, inform participants of an error, and allow them to return to the gateway page. Upon returning to the gateway, time was restarted. The test monitors were also responsible for recording difficulties encountered by users while completing tasks (e.g. select a link that does not lead to the appropriate resource) under the comments section of the Data Collection Sheet (See Appendix E).

Following each task, participants were reminded to place their hands on the mouse and move it to the bottom centre of the screen to give each task the same staring point in the time trial. This was repeated for all subsequent questions. A second questionnaire concerning usability of the gateway was presented to the participants following the successful completion of all 10 tasks (See Appendix F). Finally, a feedback letter was given to each participant (See Appendix G).

Measures:

Measures obtained from keystroke level analysis were used as benchmarks to compare the times recorded in user testing. Keystroke level analysis is essentially a sub-component of Goal Operator Method Selection (GOMS) rules and provides a specific level of detail. GOMS is a task analysis technique that allows the designer to know how quickly an experienced user can perform a specific sequence of events. Components of a computer task called "operators" include: striking a key, pointing with a mouse, homing hands on keyboard or other device, mentally preparing and response by system. Each of these operators has associated efficient times that were derived from experiments and they are summarized in a chart that was created by Card et al. (1983). This chart has become the reference tool used by experimenters who are applying the keystroke level analysis.

The purpose of these analyses is to predict speed of execution and apply rules for error-free operation or task completion. The method involves dividing tasks into their components and assigning execution times measured in seconds to each. The end result produces a quantifiable number, which represents the estimated performance time of the whole operation. This number can subsequently be compared to other tasks to show which is more efficient. For the purposes of this project, the keystroke analysis was completed on 10 task questions, and compared to the times the participants actually required to answer the questions.

This model was designed specifically for human computer interactions (HCI), which is ideal for analysing computer tasks. This method is used to analyse the time an experienced user requires to complete a specific task. However, conduction of the usability test was completed on users who were not expert users of the gateway. Therefore, the times recorded for the observed testing and theoretical key analysis were quite different. This will be explained further in the results section.

When completing the keystroke level analysis, it was assumed that users were average typists, which corresponds to a time of 0.20 sec to type one character. It was further assumed that the mouse pointer was positioned at the bottom centre of the screen prior to beginning the task. Therefore, the time normally required to home hand to mouse was eliminated.

There were a number of special rules determined by the team for the timing of tasks. Firstly, the timing of the task was not to begin until the experimenter completed reading the question. Secondly, time was stopped temporarily following an error and only restarted once they returned to the gateway page to continue with the completion of the task. It is important to note that timing may have been affected by participants' request to have questions restated during the course of the task. It was found that during usability testing, users would place the cursor on a link and ask the experimenter whether it was correct, rather than actually clicking off the gateway. This elongated the time required to complete the task and was inconsistent among participants. The script instructions were not specific enough. Users were unaware that clicking off the gateway indicated a successful completion of the task. Also, specific phrasing of usability questions may not have been clear enough for the participants. As a result, some were confused as to the specific goal.

Intent of Method

The purpose of the t test for the two independent means is to help the designer decide whether an observed difference between two means arose by chance or represents a true difference. This decision cannot be made with certainty; it is probabilistic. The problem, then, is to determine the probability of observing the difference between the means of the two groups under the assumption that the null hypothesis is true (i.e., there is no difference between the two groups) (Shavelson, 1996).

Method

Usability testing was performed on a combination of experienced and advanced users. Their times were collected and summarized. Keystroke level analyses were performed on the same questions to represent the expert population and to set benchmarks. (See Appendix L for results).

An interactive statistical web site was used to perform the t-test. It performed the t test on two sets of data to see if there was a significant difference between the means of them (GraphPad Software, Inc., 2001). Our two data sets were taken from the series of lab-based usability testing and the keystroke level analysis of expert users. The data was tabulated and the mean, probability value and standard error of each task were recorded. Confidence intervals and t critical values based on the class methods were also performed. Please refer to Table 1 for a summary of results and t-test.

In order to decide whether or not to reject the null hypothesis, a comparison between the observed t value and the critical value was done (Shavelson, 1996). The t-critical value designates the points in the t distribution beyond which differences between the means are unlikely to arise under the null hypothesis. Since there was a different critical value of t for each degree of freedom, the next step was to determine the number of degrees of freedom.

The critical value of t for the calculated degrees of freedom at a specified level of significance was determined by looking up the value on a table. It was assumed that a two-tailed test at a = 0.05 was planned before the study was conducted. Table C from Shavelson (1996) was used to determine this critical value. When all values were obtained, a comparison between the observed and critical values were conducted and conclusions based on these values were made.

The results from the usability testing based on the t-test showed that the keystroke level analysis alone, is not a good indicator of user activities. The keystroke level analysis gave each task the same amount of time to leave the Gateway page. It was unable to predict the varying mental load it would take to complete each task. This was also due partially to the method not being able to represent the population that was tested. All lab-based usability testing was conducted on undergraduate students so a task such as finding alumni borrowing privileges may be more difficult. The times recorded for the actual testing revealed that certain questions required more time to complete than the benchmarks from the keystroke level analysis. For example in task 6, where the user had to search for an electronic dictionary, they took long amounts of time reading over the entire site to find an appropriate link off of the Gateway. It was assumed that these questions required more time because users were unable to quickly identify the appropriate links. For this case, the wording of the links will require further research and study to use language that is appropriate to the target users. Thus, the designers focused on these problematic areas to suggest final recommendations to mediate these problems on the last prototype.

The chi-square analyses illustrated that there were significant differences between the observed and expected frequencies with respect to user ratings of usability for all elements with the exception of keywords. Thus, the majority of the frequencies observed were significantly different from the expected values. So in terms of user friendliness, the prototype gateway scored well for users that were navigating the site. They enjoyed the graphics and the layout of the page. However, the wording used in our prototype did not show a tendency towards user friendliness and this has been identified as a problem area to address for future work.

Final Design Specifications

Based on the findings of the user testing, numerous changes were made to the medium fidelity prototype. It was evident that many changes must to be made to the wording and structure of the headings so that the site can be navigated with more ease.

The issues that were raised by the client have been addressed over the course of this project. In the library's current form, it has been found that the mouse over interface is difficult to use in its current form as a clickable interface. The use of mouse overs as an informative window provides information to novice users that are learning to use the site.

The white space and graphics were not suitable for the current UW Library and generated negative responses from undergraduates. These issues were corrected with more use of the space and a graphic representative of our university.

The final headings and underlying links can be found in table 7 below. Some of the changes made from the previous prototype to the final prototype will be explained below the table.

Table 7: Final Categories

The link "connect from home" was moved from the Tools for You heading to the How Do I...? heading. This was done because "connect from home" is a task that people usually want to find help on so the most intuitive place to look for this information would be under the How Do I...? heading.

It was decided to change the name of the link "Staff/Admin" to "Contact Library Staff" because it was found that most users had a difficult time trying to find contact information for their Liaison Librarian. The design team felt that new title of the link is more intuitive and should result in less confusion. Further testing on this title should be conducted to prove its effectiveness.

Under the Find Books, Journals, Data... heading, the link to "By Subject" was changed to "Research Guides". This was done because almost every user that was asked what they thought "By Subject" meant, had the wrong interpretation. The term "Research Guides" is more descriptive and depicts more accurately what the link actually represents.

The wording for the Quick Search was also changed. The word "In" was changed to "By". This makes the sequence flow better and makes the users less confused as to the purpose of the quick search. This problem was addressed because during the user testing, a lot of participants used the quick search to find things that were not related to that task.

The How Do I...? button at the top of the page was removed because of redundancy. There is also a heading How Do I...? on the left side of the page so it was determined that one of them needed to be removed. The button was removed instead of the link because there is no mouse over functionality in the button so it is not possible with the current design to give a description of what can be found within that heading.

Some additional changes that were made is that all the text found on the page is now displayed using the Arial font. This was done to keep a consistent font for the web page as well as to use a heading font for our links. Also the colour of the text links was changed from the default colour (blue) to yellow. This was done because there were a lot of complaints from the users that it was very difficult to read to quick links on the right side of the page because of the colour of the links. The colour yellow was chosen because it contrasts with the black background and it is consistent with the UW colours that have been used throughout the site. The visited link colour was changed from the default (purple) to orange with the same reasoning in mind.

The final version of the prototype can be seen below in Figure 4. It was discussed in Interim Report #1 that the inconsistency of the pages found from the web gateway was problematic and that it would be ideal if they could be redone. Figure 5 shows what the secondary pages will look like. This layout will remain the same for all of the pages that can be accessed from library gateway.

It should be clarified that the links displayed in the bubble are not clickable. The reasoning behind this is that the user needs to click on the main heading that he/she wants and from there they will get a description of where each link takes you. This can be seen in Figure 5.

Figure 4: Final Design

Figure 5: Final Design of the Sub-page

Recommendation can be suggested regarding usability testing as well as web page effectiveness. The usability of the final prototype must be further investigated prior to implementation. Testing should be extended to all users including faculty, librarians, graduate students, etc. rather than focussing solely on undergraduate students. In addition, user sample size should be increased in order to increase the generalizability of the statistical analysis. Another issue that should be looked at is possibly creating a search engine that searches multiple databases. Currently the search engines are divided in categories such as e-journal, journals, books, etc. Search efficiency could be improved if these separate engines were combined into a global search database containing relevant fields.

References

Usability.gov: Usability Basics http://usability.gov/basics/index.html March 24, 2001

Appendix A: Office of Research Ethics forms

UNIVERSITY OF WATERLOO

OFFICE OF RESEARCH ETHICS

APPLICATION FOR ETHICS REVIEW OF RESEARCH INVOLVING HUMAN PARTICIPANTS

IN UNDERGRADUATE COURSE PROJECTS

1. Title of Project: Usability Testing of UW Library Gateway Web Page

2. Faculty Supervisor(s) Department Ext: e-mail:

Carolyn MacGregor Systems Design Eng 2897 cgmacgre@engmail

3. Student Investigator(s) Department e-mail: Local Telephone Number:

SD 348 Winter 2001 students - see attached list

4. Level of Project: Undergraduate Course Specify course and number: ___SD 348_____

5. Indicate the anticipated commencement date for this project: __March 22, 2001___

Indicate the anticipated completion date for this project: __March 30, 2001____

We have been working on the redesign of the UW Library Gateway page in conjunction with the UW Library Community Needs Assessment Group (CNAG). The members of CNAG serve as clients for the 6 teams of students (5-6 students per team). All teams are working on the same assignment - the redesign of the UW Library Gateway page through the use of user-centred design methods. The methods and redesign activities to date have involved the students and the CNAG members. As the final phase of the design projects, the students will be carrying out formal "lab-based" usability testing of their final designs and the current UW Gateway page.

The project does not involve a research question per se. The overall objective of the usability testing is to have the students experience a more formal method of evaluating their designs. The intention of the project is to generate recommendations for alternative designs for the UW Gateway page that can then be more rigorously tested by CNAG.

7. Methodology/Procedures

a.Which of the following procedures will be used? Provide a copy of all materials to be used in this study.

[ ] Survey(s) or questionnaire(s) (mail-back) Are they standardized? All [ ] Some [ ] None [ ]

[ ] Survey(s) or questionnaire(s) (in person) Are they standardized? All [ ] Some [ ] None [ ]

[X ] Computer-administered task(s) or survey(s) Are they standardized? All [ ] Some [ ] None [ X ]

[ X ] Interview(s) (in person)

[ ] Interview(s) (by telephone)

[ ] Focus group(s)

[ ] Audiotaping

[ ] Videotaping

[ ] Unobtrusive observations

[ ] Analysis of secondary data set (no involvement with human participants)

[ ] Analysis of human tissue, body fluids, etc. only

Other (specify)

b. Provide a brief, sequential description of the procedures to be used in this study.

All users will be asked to read and sign the Information/Consent Letter. See Appendix A.

All users will be asked to fill out a brief Background Questionnaire concerning their familiarity with the main UW libraries and the UW Library website (e.g. frequency of visits) and web pages in general. See Appendix B.

Test monitors will read from a set script and each user will be asked to perform a set of tasks related to library activities using either the current UW Library Gateway Page or the redesigned Gateway page. Users will be reminded that the objective of the study is to test the usability of the design (and not to test the skills of the user). See Apprendix C.

Test monitors will time how long it takes the users to complete the tasks, and will record when users have encounter difficulties in completing tasks (e.g. select a link that does not lead to the appropriate resource).

See Appendix D.

Participants will be asked to fill out a Usability Questionnaire once the tasks have been completed.

See Appendix E.

All users will be given a feedback letter. See Appendix F.

8. Participants Involved in the Study

Non-UW Participants: [ ] Children

[ ] Adolescents

Other (specify) _________________________________________

For the purpose of this design project, participants can be anyone who is a current UW student. In order to test the robustness of their designs, teams will be encouraged to recruit a range of participants (e.g. mix of males and females, undergrads and grads, different disciplines). The only requirement is that the participant has used web pages before.

c. How many participants are expected to be involved in this study?6 design groups X 6-10 participants

1. Recruitment Process and Study Location

a. From what source(s) will the potential participants be recruited?

[ X ] UW undergraduate and/or graduate classes

[ ] UW Psychology Research Experiences Group

[ ] Other UW sources (specify) _______________________

[ ] School Boards (not including local school boards)

[ ] Kitchener-Waterloo Community

[ ] Agencies

[ ] Businesses, Industries

Since the majority of the recruitment will be done directly by the SD 348 students, a short "email" message has been prepared that they can send to personal contacts. See Appendix G.

c. Where will the study take place? [ X ] On campus Location __EL 108 (computer lab)

If Yes, provide details:

See Appendix F

The participants will experience usability testing and their feedback and efforts will contribute to the evaluation and redesign of the UW Library Gateway Page.

Ultimately usability improvements to the UW Library Gateway Page will allow the larger community greater ease in accessing UW Library information and resources.

13. POTENTIAL RISKS FROM THE STUDY

[ X ] No known or anticipated risks

Explain why no risks are anticipated:

Participants will be asked to carry out brief tasks using a webpage. All participants will be familiar with using webpages. Participants will be reminded that their performance is being observed to test the effectiveness of the designs - not to test the skills of the users.

[ ] Minimal risk

Description of risks:

[ ] Greater than minimal risk

Description of risks:

14. INFORMED CONSENT PROCESS

15. ANONYMITY OF PARTICIPANTS AND CONFIDENTIALITY OF DATA

Information/consent forms with participant names will be kept in a file separate from the data collection forms to protect anonymity. Once data collection is finished and aggregated raw data (i.e. consent forms, data collection sheets and questionnaires) will be turned over to Prof MacGregor.

[ ] Paper Records

[ ] Confidential shredding after ______ years

[ ] Data will be retained indefinitely in a secure location

[ X ] Data will be retained until completion of specific course.

[ ] Audio/Video Recordings

[ ] Erasing of audio/video tapes after ______ years

[ ] Data will be retained indefinitely in a secure location

[ ] Data will be retained until completion of specific course.

[ ] Electronic Data

[ ] Erasing of electronic data after ______ years

[ ] Data will be retained indefinitely in a secure location

[ X ] Data will be retained until completion of specific course.

[ ] Other (Provide details on type, retention period and final disposition, if applicable)

[ ] Other: _____________________________________________________________________________

INVESTIGATORS' AGREEMENT

I have read the Office of Research Ethics Guidelines for Research with Human Participants and agree to comply with the conditions outlined in the Guidelines. In the case of student research, as a Course Instructor, my signature indicates that I have read and approved the application and proposal, deem the project to be valid and worthwhile, and agree to provide the necessary supervision of the student(s).

_____________________________ _March 15, 2001________

Signature of Course Instructor Date

____________________________________ _________________________

Signature of Student Investigator(s) Date

____________________________________ _________________________

Signature of Student Investigator(s) Date

____________________________________ _________________________

Signature of Student Investigator(s) Date

FOR OFFICE OF RESEARCH ETHICS USE ONLY:

_____________________________ _________________________

Susan E. Sykes, Ph.D., C. Psych. Date

Director
Office of Research Ethics

SYDE 348 Winter 200

Instructor:

Carolyn MacGregor, Systems Design Eng, cgmacgre@engmail.uwaterloo.ca

Teaching Assistants:

Scott Anderson, Systems Design Eng, sjanders@engmail.uwaterloo.ca

Lora Bruyn, Systems Design Eng, le2bruyn@engmail.uwaterloo.ca

Class List

Student Name, Department, Email

Minh Cao, Systems Design Eng, mcao@engmail.uwaterloo.ca

Justina Chan, Systems Design Eng, jpwchan@engmail.uwaterloo.ca

Cecilia Chung, Systems Design Eng, cchung@engmail.uwaterloo.ca

Ian Dai, Systems Design Eng, y2dai@engmail.uwaterloo.ca

Tim Filier, Kinesiology, tfillier@ahsmail.uwaterloo.ca

Ryan Finnie, Kinesiology, rlfinnie@ahsmail.uwaterloo.ca

Ankur Gupta, Kinesiology, a8gupta@ahsmail.uwaterloo.ca

Phat Ha, Kinesiology, phha@ahsmail.uwaterloo.ca

Veronica Haliniak, Kinesiology, vmhalini@ahsmail.uwaterloo.ca

Danny Ho, Systems Design Eng, ycdho@engmail.uwaterloo.ca

Nadim Jamal, Systems Design Eng, njamal@engmail.uwaterloo.ca

Shawn Kavanagh, Kinesiology, smkavana@ahsmail.uwaterloo.ca

Chris Klachan, Kinesiology, cdklacha@ahsmail.uwaterloo.ca

Gerald Lai, Kinesiology, gelai@ahsmail.uwaterloo.ca

Frank Lardi, Kinesiology, flardi@ahsmail.uwaterloo.ca

Pamela Lauz, Systems Design Eng, plauz@engmail.uwaterloo.ca

Arthur Law, Systems Design Eng, alaw@engmail.uwaterloo.ca

Rhoda Lee, Kinesiology, r6lee@ahsmail.uwaterloo.ca

General Leung, Systems Design Eng, g2leung@engmail.uwaterloo.ca

Dylan Lum, Systems Design Eng, dlum@engmail.uwaterloo.ca

Jasdeep Madpuri, Kinesiology, jkmadpur@ahsmail.uwaterloo.ca

Carrie Ng, Kinesiology, cgng@ahsmail.uwaterloo.ca

Scott Nisbet, Kinesiology, ssnisbet@ahsmail.uwaterloo.ca

Barry Piquet, Systems Design Eng, brpiquet@engmail.uwaterloo.ca

Kelly Jo Shipp, Systems Design Eng, kjshipp@engmail.uwaterloo.ca

Robert Snow, Systems Design Eng, rjsnow@engmail.uwaterloo.ca

Greg Stefan, Kinesiology, gastefan@ahsmail.uwaterloo.ca

Melissa Thomas, Kinesiology, m2thomas@ahsmail.uwaterloo.ca

Julia Thompson, Kinesiology, je4thomp@ahsmail.uwaterloo.ca

Anna Tran, Kinesiology, a4tran@ahsmail.uwaterloo.ca

Justine Yau, Kinesiology, jyau@ahsmail.uwaterloo.ca

Simon Yu, Kinesiology, s2yu@ahsmail.uwaterloo.ca

Appendix B: Information/Consent Form

SD 348 Winter=01 Team Project

Information Letter

RESEARCHERS: Student Members of SD 348 (User-Centred Design), in co-operation with

Prof. Carolyn MacGregor (Systems Design Engineering) and

the members of UW Library Community Needs Assessment Group (CNAG)

STUDY OBJECTIVES:

The class members of SD 348, along with course instructor Prof. Carolyn MacGregor, are working with CNAG to help improve the design and layout of the University of Waterloo's Library Gateway (main) web page.

The main objective of this usability testing is to evaluate the logic and layout of the current UW Library Gateway web page and to compare it with some proposed redesigns.

TASKS:

All participants will complete a Background Questionnaire. The purpose of this questionnaire is to collect general information on the demographics and computer & library experience of the participants involved in this phase of the usability testing.

Following the questionnaire, you will be asked to carry out a set of specific tasks. At the end of the tasks, we will ask you some general questions on your opinions of how user-friendly you found the system.. You may leave unanswered any questions you prefer not to answer.

Keep in mind that the testing is not an evaluation of how well you might use the system, but whether the web pages are user-friendly at this phase in the design development.

TIME COMMITMENT:

This study will require approximately 20-30 minutes of your time. Your participation is strictly voluntary.

RISKS & BENEFITS:

There are no known risks to yourself. The benefits of participating in this study is that you will be contributing to the development of a more user-friendly design for the UW Library Gateway web page

RIGHT TO WITHDRAWAL:

You may withdraw from this study at any time by advising the researcher of your decision. At the time of withdrawal, all data collected during your session will be destroyed.

CONFIDENTIALITY:

All data collected will be kept confidential, and you will not be personally identified in any reports. A participant code will be used on all data sheets instead of your name. The only people who will have access to the original data will be the researchers. All data will be destroyed at the end of the course requirements (April 2001).

ETHICS REVIEW:

If you have any questions about participation in this study, please feel free to ask the researchers. If you have additional questions at a later date, please contact Professor MacGregor at 888-4567 ext 2897.

This project has been reviewed by, and received ethics clearance through the Office of Research Ethics at the University of Waterloo. If you have any questions or concerns resulting from your participation in this study, please contact Dr. Susan Sykes at 885-4567 ext. 6005.

Consent of Participant

I have read and understood the information presented in the participant information letter and understand the procedures and risks involved in the study. I have received satisfactory answers to my questions related to this study. I understand that this project has been reviewed by, and received ethics clearance through, the Office of Research Ethics at the University of Waterloo. I further understand that if I have any questions or concerns resulting from my participation in this study, I may contact the Director, Office of Research Ethics, at (519) 888-4567 ext. 6005. I am aware that I may withdraw from the study at any time without penalty. With full knowledge of all foregoing, I agree, of my own free will, to participate in this study.

_____________________________

Print Name Signature of Participant Dated at Waterloo, Ontario Witnessed

Do you wish to receive feedback on the results of this study? ___ No ___ Yes

If yes, please include an email or regular mailing address where a summary of the results can be sent?

Email: ________________________

Appendix C - Background Questionnaire

USABILITY TESTING OF UW LIBRARY GATEWAY WEB PAGE.

BACKGROUND QUESTIONNAIRE

General Questions:

Gender: ___ Male ___ Female Age: ____

Program: ___________________ Year: ____ Co-op: ___ Yes ___ No

On a scale of 1 to 5, how familiar would you say you are with everyday computing activities

(e.g. word processing, email)?

On a scale of 1 to 5, how familiar would you say you are with the use of web-pages for on-line activities (e.g. searching for information)?

On a scale of 1 to 5, how familiar would you say you are with the design and creation of web-pages?

On a scale of 1 to 5, how familiar would you say you are with UW's Library web-pages?

On a scale of 1 to 5, how familiar would you say you are with UW's Library services?

Appendix D - Test Monitor Scripts

TEST MONITOR SCRIPTS

Note to Test Monitors:

Please read the text in Times-Roman (this font) out loud to each participant. The text in capitals (Arial) are instructions to you and should not be read out loud.

LIST OF USER-TESTING TASKS:

Instructions to Participant, Information/Consent & Background Questionnaire (5 minutes)

Library Tasks using current design or redeisgn (10-20 minutes)

Usability Questionnaire (5 minutes)

EQUIPMENT YOU NEED TO BRING TO TESTING SESSIONS:

• testing script
• data collection forms
• a watch to time events
• clipboard (or something to write on)
• extra paper for making notes

Step 1: INSTRUCTIONS TO PARTICIPANTS.

NOTE: MAKE SURE THAT WINDOW FOR WEB PAGE IS MINIMIZE SO THAT PARTICIPANT CANNOT SEE IT UNTIL THE TRIALS ARE READY TO BEGIN

Thank you for agreeing to participate in our study entitled AUsability Testing of the UW Library Gateway Web Page@.

This study is being carried out by the class members of SD 348 (User-Centred Design) as part of our course requirements. In order to make sure that all participants receive the same information, I am going to read to you from this script.

We will start by going over the information letter for this study. It will explain the objectives of this study and the tasks that we will ask you to perform.

GO OVER INFORMATION LETTER, AND ASK IF PARTICIPANT HAS ANY QUESTIONS?

ASK PARTICIPANT TO READ OVER AND SIGN CONSENT FORM.

Step 2: BACKGROUND QUESTIONNAIRE

Before we have you use the UW Library Gateway Web Page we would like to ask you some general questions about yourself and your computing experience.

ASK STUDENT TO FILL IN THE BACKGROUND QUESTIONNAIRE.

Now for the main part of the study.

Step 3: LIBRARY TASKS

Before we start, I need to remind you that we are testing the usability of the web page design and not your ability to use the system. I will be taking notes as you work through the tasks and recording how long it takes someone to find the appropriate links and where people may have difficulties using the web page. This will help us to improve upon our design before our presentations next week.

I will be asking you to work through a series of tasks that people normally associate with libraries. We will be going through each task one right after the other and then I will give you an opportunity to comment about the web page and tasks once they are all done.

For each task, you will start with your hand on the mouse and the cursor at the bottom centre of the screen. I will then read the task out loud and once I am finished you are to try to find the appropriate link on the web page as quickly as possible. I will tell you when you have found the appropriate link which will end that task. I will then remind you to move the cursor to the bottom centre of the screen before I read the next task.

• BRING UP WINDOW AND ASK PARTICIPANT TO PLACE HAND ON MOUSE AND MOVE CURSOR TO BOTTOM CENTRE OF SCREEN.

• READ FIRST TASK FROM SHEET.

• START TIMER.

• STOP TIMER WHEN PARTICIPANT HAS CLICKED ON APPROPRIATE LINK THAT MOVES USER OFF OF THE GATEWAY.

• NOTE ANY PROBLEMS ON OBSERVATION SHEET BEFORE MOVING TO NEXT TASK.

• REMIND PARTICIPANT TO MOVE CURSOR TO BOTTOM CENTRE AND KEEP HAND ON MOUSE WHILE YOU READ NEXT TASK OBJECTIVE.

ONCE ALL TASKS ARE COMPLETED, ASK PARTICIPANT TO FILL OUT USABILITY QUESTIONNAIRE.

Step 4: USABILITY QUESTIONNAIRE

As a final task we would like you to answer some questions about the usability of the UW Library Gateway Web-Page design that you just used.

Appendix E - Data Colletion Sheet

Test Monitor: ________________ Participant #: ________________

RECORD TIME AND OBSERVATIONS

APPENDIX E (con't) Data Collection Task List

Appendix F - Usability Questionnaire

1. On a scale of 1 to 5, how USER-FRIENDLY would you say the web-pages were in terms of conveying which actions you needed to perform to complete the following tasks?

   	Not at all user-friendly			Very User-friendly
   Key words	9	9	9	9	9
   Major Links	9	9	9	9	9
   Graphics	9	9	9	9	9
   Navigation Bars	9	9	9	9	9
   Overall Layout	9	9	9	9	9

2. A) In terms of overall usability, how would you compare this web page with the current UW Library Gateway Page

   9	not applicable, (haven=t used current UW Library Gateway Page)	--go to #3
   9	much better than current UW Library Gateway Page	-- go to #3
   9	marginally better than current UW Library Gateway Page	-- go to #3
   9	about the same as current UW Library Gateway Page	-- go to #3
   9	marginally worse than current UW Library Gateway Page	(go to # 2b)
   9	much worse than current UW Library Gateway Page	(go to #2b)

   B) In your opinion, what would you say are the major problems with the new design that makes it

   worse than the current UW Library Gateway Page?

3. Do you have any additional comments or feedback on the web pages you used today?

Appendix G - Feedback Letter

USABILITY OF THE UW LIBRARY GATEWAY WEB PAGE

March 22, 2001

Dear Participant:

Thank you for helping us with our usability study as part of our course requirements for SD 348 (User-Centred Design). The class has been working with the UW Library's Community Needs Assessment Group to help improve the usability of UW's Library Gateway Web Page. UW would like to make sure that its library resources and services are accessible and easy to find for UW students and faculty, UW alumni, and the larger community. Improving the usability of the library web pages will help them to meet that goal.

There are six different teams in our class. Each team has been working on an alternative design for the current UW Library Gateway Web Page. We will be using the information that we have gathered from you along with the other participants to help improve upon the usability of our web page designs. As stated in the information letter, the data from participants will be analysed as a group and you will not be identified personally.

As a class we have been following a user-centred approach to design. Each team has developed their own prototype of the UW Library Gateway that has evolved over the course of the term through usability analysis and testing. Our class will be giving a presentation of the evolution of our web designs on

Tuesday, March 27, 2001at 9 am - 10:30 am in E2 1303 B.

You are welcome to attend to see how your input may have helped with the development of the final designs.

If you would like further information about this project, please contact Prof. Carolyn MacGregor at 888-4567 ext 2897 or at cgmacgre@engmail.uwaterloo.ca

If you have concerns regarding your participation in this study, please contact Dr. Susan Sykes at the Office of Research Ethics, 888-4567 ext 6005.

Thank you again for your participation.

The Students of SD 348.

Appendix H: Email "flyer"

WANTED: VOLUNTEERS FOR Usability Testing of UW Library Main Gateway Web Page

ARE YOU A UNIVERSITY OF WATERLOO STUDENT?

DO YOU HAVE 20-30 MINUTES TO TEST THE USABILITY OF THE UW LIBRARY GATEWAY WEB PAGE???

The class members of SD 348, along with course instructor Prof. Carolyn MacGregor, are working with the UW Library Community Needs Assessment Group to help improve the design and layout of the UW Library Main Gateway Web Page.

UW Library has lots of great resources on-line and wants to make sure that students can easily find that information.

This is your chance as a UW student to help with the preliminary user testing of the current and new web-page designs.

If you are interested in participating, please contact [student puts email address here]

or if you have further questions please contact

Prof. MacGregor at cgmacgre@engmail.uwaterloo.ca

This project has been reviewed by, and received ethics clearance through, the Office of Research Ethics at the University of Waterloo. If you have any questions or concerns resulting from your participation in this study, please contact the Director, Office of Research Ethics, at (519) 888-4567 ext. 6005.

Appendix I - Neilsen's ten recommended heuristics (Nielsen, 2001):

Nielsen's ten recommended heuristics are as follows:

1. Visibility of system status
2. Match between system and the real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognise, diagnose, and recover from errors
10. Help and documentation

Appendix J: Hierarchical Task Analysis Details

This appendix details the Hierarchical Task Analysis procedure for UW Library web site.

Table 8: Task 1 - Find in the book Alias Grace, by Margaret Atwood at the UW library

Table 9: Task 2 - Find out if the University Map and Design Library is open in weekends

Table 10: Task 3 - Find the librarian for Systems Design Engineering

Table 11: Task 4 - Find a database in which to locate articles on Anthropology subjects

Table 12: Task 5 - Get a copy of Stanton in a remote library

Appendix K: Task Allocation

Interim report #1 write-up (introduction, ISPS, future considerations); final editing of Interim report #1; designed low fidelity Prototype #2 and medium fidelity prototype, participated in Card Sorting; conducted heuristics and informal user testing; write-up of interim report #2; real user testing and write-up of interim report 3 (Introduction to problem; revised ISPS; phase 1 methods)

By Justina Chan: _________________________

Interim report #1 write-up(heuristics evaluation and cognitive walkthrough methods and results, Gantt chart and usability concerns); designed low fidelity Prototype #1 and medium fidelity prototype, participated in Card Sorting, conducted Heuristics and informal user testing along with write-up of interim report #2; real user testing; interim report #3 (phase 3 methods, results and discussion; final proof-reading), Presentation

By Frank Lardi: _________________________

Interim report #1 (Competitive analysis, draft design sketch, Gantt chart and usability concerns); designed low fidelity Prototype #1 and medium fidelity prototype, participated in Card Sorting, conducted Heuristics and informal user testing along with write-up of interim report #2; real user testing; report 3 (Phase 2 methods, summary of client meetings, team learning)

By Jasdeep Madpuri: _________________________

Interim report #1 (heuristics evaluation and cognitive walkthrough methods and results, Gantt chart and usability concerns); designed low fidelity Prototype #2 and medium fidelity prototype, participated in Card Sorting, conducted Heuristics and informal user testing along with write-up of interim report #2; real user testing; report 3 (phase 3 methods, results and discussion; final proof reading) Presentation

By Carrie Ng: _________________________

Implemented functional prototypes; interim report #1 write up (HTA, usability concerns; design alternatives); participated in Card Sorting, implemented medium fidelity prototype and formatting of reports; phase 3 results and discussions; executive summary

By Nadim Jamal: _________________________

Implemented functional prototypes; interim report #1 write up (HTA, design alternatives); participated in Card Sorting and implemented medium fidelity prototype; real user testing; report 3 (testing results and discussions); Presentation

By Arthur Law: _________________________

Appendix L: Keystroke Analysis for Data Collection Task List (Version A)

1. Where would you go to find general help with a research topic in your department?

   Table 13: Experienced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Scan the top bar on the page (5 word chucks)	5.00 s
   3.	Scan the Left Side Links to "Tools For You" (5 word chunks)	5.00s
   4	Point to link (Tools for You)	1.10 s
   6	Click on link (Tools for You)	0.20 s

   TOTAL 12.65 s

   Table 14: Advanced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Scan the top bar on the page (5 word chucks)	5.00 s
   3	Scan the Left Side Links to (5 word chunks)	5.00 s
   4	Scan Right Side Column of quick links (21 word chunks)	21.00 s
   5	Point to quick link (Reference Guide)	1.10 s
   6	Click on quick link (Reference Guide)	0.20 s

   TOTAL 33.65 s

2. How do you get an article from a journal that is available at WLU?

   Table 15: Experienced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Scan the top bar on the page (5 word chucks)	5.00 s
   3.	Scan the Left Side Links to "Links to Other Libraries" (5 word chunks)	5.00s
   4	Point to link (Links to Other Libraries)	1.10 s
   6	Click on link (Links to Other Libraries)	0.20 s

   TOTAL 12.65 s

   Table 16: Advanced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Point to link (Combo box)	1.10 s
   3	Mentally prepare (as reads menu)	1.35 s
   4	Point to selection (ILL Form)	1.10 s
   5	Click on link (ILL Form)	0.20 s

   TOTAL 5.10 s

3. Can Alumni borrow books from UW?

   Table 17: Experienced User/Advanced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Scan the top bar on the page (5 word chucks)	5.00 s
   3.	Scan the Left Side Links to "About the UW Library" (5 word chunks)	5.00s
   4	Point to link (About the UW Library)	1.10 s
   6	Click on link (About the UW Library)	0.20 s

   TOTAL 12.65 s

4. What are the hours for the University Archives?

   Table 18: Experienced User/Advanced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Scan the top bar on the page (5 word chucks)	5.00 s
   3.	Scan the Left Side Links to "About the UW Library" (5 word chunks)	5.00s
   4	Point to link (About the UW Library)	1.10 s
   6	Click on link (About the UW Library)	0.20 s

   TOTAL 12.65 s

5. What books are available at the Kitchener Public Library?

   Table 19: Experienced User/Advanced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Scan the top bar on the page (5 word chucks)	5.00 s
   3.	Scan the Left Side Links to "Links to Other Libraries" (5 word chunks)	5.00s
   4	Point to link (Links to Other Libraries)	1.10 s
   6	Click on link (Links to Other Libraries)	0.20 s

   TOTAL 12.65 s

6. Does the library have any electronic dictionaries?

   Table 20: Experienced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Scan the top bar on the page (5 word chucks)	5.00 s
   3.	Scan the Left Side Links to "Tools For You" (5 word chunks)	5.00s
   4	Point to link (Tools For You)	1.10 s
   6	Click on link (Tools For You)	0.20 s

   TOTAL 12.65 s

   Table 21: Advanced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Scan the top bar on the page (5 word chucks)	5.00 s
   3	Scan the Left Side Links (5 word chunks)	5.00 s
   4	Scan Right Side Column of quick links (21 word chunks)	21.00 s
   3	Point to quick link (Reference Guide)	1.10 s
   4	Click on quick link (Reference Guide)	0.20 s

   TOTAL 33.65 s

7. Does UW library have a copy of Andrew Pyper's Lost Girls?

   Table 22: Experienced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Scan the top bar on the page (5 word chucks)	5.00 s
   3.	Scan the Left Side Links to "Find books, journals, data..." (5 word chunks)	5.00s
   4	Point to link (Find books, journals, data...))	1.10 s
   6	Click on link (Find books, journals, data...))	0.20 s

   TOTAL 12.65 s

   Table 23: Advanced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Scan the top bar on the page (5 word chucks)	5.00 s
   3	Point to quick link (TRELLIS)	1.10 s
   4	Click on quick link (TRELLIS)	0.20 s

   TOTAL 7.65 s

   Table 24: Advanced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Point to quick link (Search textbox)	1.10 s
   3	Click on quick link (in Search textbox)	0.20 s
   4	Home hands to keyboard	0.40 s
   5	Mentally prepare (thinking of what to type)	1.35 s
   6	Type 10 characters (Lost Girls)	10*0.2=2 s
   7	Home hand to mouse	0.40 s
   8	Point to button (Go)	1.10 s
   9	Click mouse	0.20 s

   TOTAL 8.10 s

8. Where can you find instructions for connecting from home?

   Table 25: Experienced User/Advanced Users

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Scan the top bar on the page (5 word chucks)	5.00 s
   3.	Scan the Left Side Links to "Tools For You" (5 word chunks)	5.00s
   4	Point to link (Tools for You)	1.10 s
   6	Click on link (Tools for You)	0.20 s

   TOTAL 12.65 s

9. Where would you go to renew books online?

   Table 26: Experienced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Scan the top bar on the page (5 word chucks)	5.00 s
   3.	Scan the Left Side Links to "Tools For You" (5 word chunks)	5.00s
   4	Point to link (Tools for You)	1.10 s
   6	Click on link (Tools for You)	0.20 s

   TOTAL 12.65 s

   Table 27: Advanced User

   Steps	Operators	Time
   1	Mentally prepare (after hearing sentence)	1.35 s
   2	Point to quick link (Renewals)	1.10 s
   3	Click on quick link (Renewals)	0.20 s

   TOTAL 2.65 s

10. What is the contact information for the Liaison Librarian assigned to your home department?

    Table 28: Experienced User

    Steps	Operators	Time
    1	Mentally prepare (after hearing sentence)	1.35 s
    2	Scan the top bar on the page (5 word chucks)	5.00 s
    3.	Scan the Left Side Links to "About the UW Library" (5 word chunks)	5.00s
    4	Point to link (About the UW Library)	1.10 s
    6	Click on link (About the UW Library)	0.20 s

    TOTAL 12.65 s

    Table 29: Advanced User

    Steps	Operators	Time
    1	Mentally prepare (after hearing sentence)	1.35 s
    2	Scan the top bar on the page (5 word chucks)	5.00 s
    3	Point to quick link (TRELLIS)	1.10 s
    4	Click on quick link (TRELLIS)	0.20 s

    TOTAL 7.65 s

Appendix M: Summary of Client Meetings

Meeting #1

Date: January 22, 2001
Time: 2:00 pm
Location: Dana Porter
Present: Amos Lakos, Justina Chan, Arthur Law, Nadim Jamal, Frank Lardi, Jasdeep Madpuri, Carrie Ng,

• Amos reports the following
• He's been pushing for usability testing project for a while because he wants to keep library web site competitive
• Other libraries have done usability testing
• Outcomes will depend on : testers, testees and set up
• He doesn't want to tell us the weaknesses/strengths because that could bias our interpretation of the problems
• Present web site was built by librarians and committee, thus not user-centred
• History of web site design:
  • Originally Trellis was part of Guelph and Laurier as well
  • A company was hired to do this job but there were politics
  • Caused problems between libraries, which is why Waterloo had to develop their own gateway
• Main focus is gateway, but also navigation problems
• Look at University of Washington's web site for consecutive secondary pages
• Specific items he wants addressed in the project:
  • Reduce the number of clicks
  • Look at dead space
  • Where does your eyes go first?
  • Does language/text make sense?
  • Mouse over vs. text version; which is better?
  • Consistent back pages

Client Needs

The following summary was put together after the first client meeting. It discusses the client's goal, expectations, requirements on project methodology, questions and future considerations.

Client Goal

The client would like the UW Library web site to become focused on customers. The primary desire is to empower undergraduate students to be able to find resources on the web site with little remediation by the librarians.

Client Expectations and Scope of Project

Study the usability and generate suggestions for improving the interface for the UW Library main page using a UCD approach.

Client Requirements on Project Methodology

• Reduce number of clicks to navigate the site.
• Verify the diction in the web site links.
• Verify mouse-over interface.
• Conduct a competitive analysis on other university library web sites.

Principle Questions from Client

1. Is the mouse-over interface the correct one to use?
2. Are the labels and the language appropriate for our user group?
3. Are the graphics appropriate for this web site?
4. Is there sufficient or too much white space in the site?
5. Is the layout of the main page appropriate for searching out information?
6. Are the navigation layers correctly organised for retrieving information?

Future Considerations

• Improve usability on the web site for faculty and staff user groups.
• Find and correct navigation problems in the rest of the Library site.
• Develop a "My Library" type of portal.
• Design consistency guidelines for the pages on the UW Library site.

Meeting #2

Date: February 9, 2001
Time: 1:30pm
Location: Dana Porter Library
Present: Amos Lakos, Arthur Law, Nadim Jamal, Frank Lardi, Jasdeep Madpuri, Carrie Ng
Absent: Justina Chan

• Amos reports the following
• Would like to sit in on usability testing
• Suggests that no more than 2 or 3 people should be observing during the testing
• He would like to have an audio recording during the testing for two reasons
  1. For proof that there is a problem
  2. For back up in case of failure to record during observations
• The top priority web usability questions according to him are:
  1. How would you find if the library has Margaret Atwood's Alias Grace?
  2. Is the University Map and Design Library open on weekends?
  3. I need to contact the librarian for my department - how do I find his/her phone or email?
  4. Where can I find a database in which to locate articles on Anthropology
  5. I need to find a book that is not available in our catalogue - how can I get a copy from a remote library
• Gave us 2 articles re: usability testing for library web pages (specifically from U of Arizona)

Meeting #3

• Original intent for this meeting with Amos was to discuss our first interim report, however, we haven't got it back and would like to review the comments by TA and Prof. MacGregor before showing to Amos
• Amos requested us to send him a softcopy of the report
• Other groups in our class have sent soft copies to their clients and they are circulating them and comparing each group's findings
• Will postpone this agenda item to next meeting once we get our report back
• Amos' availability
• Will be away on March 14-18 and March 25-April 11
• He'd like to sit in for some user testing

Meeting #4

• Discussed what has been accomplished up until now
• Amos would like to have the reports emailed to him for review
• Jasdeep will send him an email with reports 1 and 2
• Discussed where we are now and what will be happening
• Prof. MacGregor is to give us the next steps to Phase 3
• Important to note the days Amos will be away
• Will set up a meeting with him next week once we know what we will be doing

Meeting #5

• Agenda for this meeting was to perform the pilot testing with Amos
• Due to technical difficulties, we were unable to perform the testing
• Asked Amos if we could reschedule for tomorrow
• Amos agreed on Wednesday at 1:30

Meeting #6

• Prototype could not be brought up by either of us
• Could not perform pilot testing
• Amos suggested we can do it on Thursday when we do the user testing

Meeting #7

• Performed the pilot testing with Amos
• Comments made by Amos regarding the medium fidelity prototype that we had for testing:
• Need the mouse over to work
• The black background makes it difficult to read
• The quick links were not visible - font size too small
• Left bar is too big - not comfortable with the different sizes on left and right
• The page is slightly easier than the 'Find it/Get it'
• The overall usability was about the same as the current UW Library Gateway page
• Amos was somewhat impressed
• Indicated that he kept using his model of the current UW web page to look for things on the prototype

Appendix N: Details from Phase 3 User Testing

User Comments

A) In terms of overall usability, how would you compare this web page with the current UW Library Gateway Page?

B) In your opinion, what would you say are the major problems with the new design that makes it worse than the current UW Library Gateway Page?

1. The major key words don't expand to give you an overview of what is found under those key words.
2. 5) I just don't like the purple links because they are hard to see.
3. 10) Small font is hard to read. No mouse overs.

The two problems with our design in comparison to the old design were the readability of the quick links and the clarification of our major links.

The quick links are very difficult to read under our current design. The font size is smaller than 12 point, which makes it difficult to use. Also, the link colour for the quick links are blue and purple. With our design of a black background, there is insufficient contrast to read the links.

The major links have some problems with being very clear. In our final prototype, we hope to have implemented a rollover graphic that gives further detail to the major links.

C) Do you have any additional comments of feedback on the web pages you need today?

1. The graphics are great.
2. Maybe make smaller menus as you drag your mouse over a specific link.
3. Right nav bar colour of items already clicked a little hard to read.
4. Good black background, big words to indicated major links and good graphics.
5. How do I...? Should be followed by verbs or links to verbs.
6. Confusion behind quick links purpose.
7. When searching for book, couldn't search by title or author.
8. Background colours and better graphics (little dude is annoying).
9. Overall, well done. One thing... The quick links and "search in ____" combo box should relatively be in the same area sectioned off by a big box (ie graphic) to show that these search tools are the same family.
10. If the mouse over feature existed, this site would be much easier to navigate.
11. The guy in the middle wastes a lot of potentially useful space. The site map is extremely useful.
12. Navigation bar at top and links on the right hand side of the page are improvements on the current site.
13. Colour scheme needs to be improved to help the site portray a more professional atmosphere. Site lacks a sense of cleanliness, brighten up.

15. The user finds the left and right navigation bars very confusing as to where to look for information. He suggested that the links should either be task-oriented or information oriented. Mixing these two up causes confusion.

In general, the test subjects approved of the graphics on the web page. The black background was noted as something desirable. One suggestion to use graphics separate the elements in the web site will also be explored.

Implementing the java script mouse over to give help text would be a good improvement to the site and would likely improve usability for the main links.

In using the search box, more features useful to a library, such as search by author and search by title would prove a good addition.

Appendix O: Team Learning

Working as a team

During the course of this project, we believe that we have gained a considerable amount of experience from working as a team. There were many obstacles that we were required to overcome and deal with. Some of these obstacles helped us to understand the meaning of team projects and some have provided us with insights into the kinds of difficulties that we may encounter in the real world.

As a team, we found that it was very difficult to work with team members who were on different time schedules. Due to the two different program backgrounds that our team consisted of, it was difficult to schedule meeting dates and times that were agreeable by all members. We also had a problem with communication between team members. It was found that team members hesitated to discuss issues that were arising throughout the duration of the project. We believe that the lack of group cohesiveness stemmed from our reluctance to communicate problems that we individually felt was affecting the team's performance, creating problems that eventually became unmanageable.

Our team consisted of students who were different in many aspects, which was hard to work with at times. However, due to the diversity, group members possessed individual skills, talents, expertise, and experience, which were valuable to the project. We learned that these differences are an important aspect of a team and should be used to our advantage. Strengths of individuals were acknowledged and tasks distributed accordingly within the scope of the project. Areas where one member is weak, can be made up for by strengths of another group member.

What we would do differently

In order to improve the team's cohesiveness and effectiveness, we believe that each member should have practised more effective communication. Relying solely on e-mails as the main communication medium, seemed to hinder our performance overall since some individuals did not check their mail within an appropriate time frame. Communication through the additional use of a phone may have improved the response time on certain issues as would regular updates within the team. Avoidance of short notices would have provided more time for problem solving. More regularly scheduled meeting times may have forced us to communicate each week regarding the project performance. Our meetings were set on a required basis, not weekly, which may have been problematic. In addition, the group should have reflected how well we were performing and voiced our opinions on a more regular basis. Comments should have been made on both positive and negative aspects of our project throughout the term so that we could retain the strengths and improve on our weaknesses.

Expectations and team norms that were set out at the beginning of the term were not maintained. Not all group members were in attendance at all meetings and team members did not commit to the assigned deadlines.

Commenting on the diversity of our group, we feel that we should have allocated tasks in a fashion where individuals with the opposite program background worked together. For example, instead of two engineers working on making the prototypes functional, a kinesiology student should have been paired with a systems design student for that task. Each individual brings different insights to the team and instead of capitalizing on this, we created even larger walls by keeping our opinions to ourselves. By dividing the tasks in this manner, individuals would have benefited from the knowledge of others and would have given each person the opportunity to share their own. It would have increased the social interactions and created even more learning experiences. This would also have eliminated faculty stereotype within our group. Upon reflection, the team was divided between Kinesiology and Systems Design students, which compounded the barriers and issues.

Overall, our team progressed from a traditional classroom learning group to a co-operative learning group. As the project progressed, each member of the team realised his or her responsibility to the group. The team's gradual improvement in the quality of work and in the quality of learning and teamwork processes, was evident in the final phase.