Web Advisory Group

"Big Test" usability test

Nicole Hennig and the Web Advisory Group
November 2002

Purpose and goals of the test

The aim of this test was to learn how our students use our many web-based tools and interfaces together. We previously tested our web site, our catalog, and Vera as separate things, but until now, we never tested how they all work together for our users. The goal was to improve the interfaces of each, including how they work together and how they refer to each other.

SFX was under development as a new service, so this was our chance to test our preliminary design of the SFX menu as well. This was also the first time that we tested our live digital reference services, "Ask Us - Live." Though "Ask Us - Live" is no longer being offered as a service, we hope to use the data to help choose future digital reference services.

Sites tested:

• web site: http://libraries.mit.edu/
• listings of databases + e-journals:http://libraries.mit.edu/vera
• web OPAC: http://libraries.mit.edu/barton
• individual databases listed in Vera
• test version of SFX menu screens in Alternative Press Index/Firstsearch test account
• "Ask Us - Live" (live chat reference service, no longer live at MIT)
• subject pages: http://libraries.mit.edu/subjects/
• The Information Navigator: http://libraries.mit.edu/tutorials/general/

 

Test Team

Nicole Hennig, Coordinator
Darcy Duke
Stephen Skuce
Christine Quirion
Jennifer Edelman
Pam Nicholas
Kate McNeill-Harman

Who we tested

21 students total (9 undergrads + 12 graduate students)

We decided to test students only this time, as the primary users of our site. (No faculty, alumni, library staff, MIT administrators, or outside users were tested).

Length of test

1 hour (usually took 45 minutes)

Reward

$15 Amazon.com gift certificate

The Test

We created two tests, one that emphasized "known-item" searching, and one that emphasized open-ended searching for materials on particular topics. We gave half the users test A and half, Test B.

Test A: "Ask Us - Live" and searching for known items

This test included a scenario for using our live chat reference service, and 8 questions involving searching for particular articles and books from a bibliography that we created.

Download test A - PDF

Test B: Open-ended searching by topic and testing SFX

This test included 6 questions involving searching by topic. Users were asked to choose one of the topics below. The questions involved doing research as if they were writing a general research paper on the topic they chose. One of the questions involved using our SFX service in a test account and one question asked users to consult what's known as "subject pages" on our web site.

Research topics (choose one):

Forest Fires
How do forest fires effect the environment? Describe the effects of large scale wild fires in relation to other environmental factors such as air or water pollution, emission of greenhouse gases, or weather patterns.

Gene Therapy
Summarize the ethical issues related to the use of gene therapy for patient treatment. Describe some of the moral and social issues surrounding gene therapy, such as patient rights, privacy, standards of care, or eugenics.

Women as scientists or engineers
What are some barriers that have kept women from becoming scientists or engineers in the past? What is the latest thinking on how to achieve equity for girls and women in science classrooms?

Download test B - PDF