Test 1
Usability test results, March - April 2001Test 1: basic questions
March 7 - 14, 2001
We decided to do more frequent, less formal tests, focusing on finding and fixing specific usability problems. Unlike our tests done in March of 1999, these tests don't include any quantitative data, such as how long it took each person to find an answer, and how many correct answers were found. The qualitative information is much more valuable. Our latest inspiration for this is a book by Steve Krug, called Don't Make Me Think: a Common Sense Approach to Web Usability.who we tested:
• 3 undergrad students
• 4 grad students
• 1 faculty member (more faculty will be tested later)
what we focused on:
• the basic, most important tasks people should be able to do easily on our site
what we looked for:
• whether our terminology and wording makes sense to people
• whether our basic structure makes sense to people
• whether people like the site (a subjective thing, of course)
the test:
• each test was about 1/2 hour long
• volunteers were asked to find the answer to 10 questions
• they were asked to "think out loud"
• we asked a few post-test questions (for the subjective opinions)
• instructions for observers
the results
Questions:
1. Where can you find the spring term hours of the libraries?No one had a problem with this. They all found it right away.
Our interpretation: Glad this works. And since several people told us that finding our hours is one of the main reasons they use our site, we decided to also add "hours" to the "quick links" menu that will appear on many pages.2. Where would you go to find out if the MIT Libraries own a certain book?
No problem.
- either they clicked on Barton on the home page, or Search Our Collections, where they succesfully found a link to Barton.Our interpretation: Glad this works. We made sure that Barton always says "MIT Libraries' online catalog" next to it. It was good to see someone using the Search Our Collections chart and finding Barton that way as well.
3. Where can you find an online dictionary?
5 of 8 people found it
- a couple of people looked in "writing" for it
- a couple of people suggested making the Virtual Reference page more prominent (people like the page if they know about it)Our interpretation: The Virtual Reference page is listed in the Search Our Collections chart, under "dictionaries, encyclopedias and more (online versions) ...". This worked for 5 people.
Since people keep saying they like the Virtual Reference page (once they find it), we decided to link to it from more pages: 1) the home page (a sample link under Search Our Collections, 2) the Writing page and 3) Publication Types.4. Where can you find a list of databases available for ocean engineering?
No problem - they all used Vera.
- at least one user entered Vera through the link on the Barton home pageOur interpretation: Glad this works. People seemed to know about Vera.
5. Find a page that lists online journals in computer science available through the MIT Libraries?
No problem - they all used Vera.
- one person entered Vera through the link on Computer Science subject page
- one person suggested that button under subjects menu should say "display list" instead of "search"
- one person hesitated slightly about which button to click (the one under subjects, or the one after keyword)Our interpretation: Glad this works. We plan to do more testing of Vera, since it's such a frequently-used page. For now we came up with some ideas for improvements:
(We've implemented some of these only on the new Vera home page, not the live Vera).
- changed the "search" buttons to say "display list" - since there is confusion about what Vera does (it's not a gateway to cross-database searching)
-added sub-title to Vera's home page: "A listing of our databases and e-journals"
- we plan to mark the different areas of the search screen with light-colored boxes, so it's easier to see which button goes with which form elements6. Where can you find a list of links to U.S. company information?
Only 1 of 8 people found it.
- when shown where it was at the end of the test every user commented that they didn't know we had that type of info (subject guides)
- several commented that they would not come to the library web site looking for subject guides
- one user said he would go to Google for this
- one user said he would have found it if our search engine was working
- several commented that they don't think of it as a "subject"
- one user said it there should be a subject called "business & management"
- one user felt that "subjects" should be called "topics" or "fields of study"Our interpretation: Part of the reason for failure was that "U.S. company information" doesn't sound like a subject. Another part had to do with the fact that people don't know we have "subject guides." We plan to do more testing of subject guides in the future. Also, we should do more publicity on them.
- we modified the link to Subject Guides on the home page:
it used to say: • subject guides
now it says: • subjects: art, math , physics... (we chose these particular subjects only because they have short names!)7. Find out if the Libraries have the online version of the Journal of Cell Biology.
No problem (except for one user who made a typo).
- everyone used Vera successfully to find this
- one user didn't because of a typo (and didn't realize she made a typo until the test was over)
comments on Vera:
- one user (faculty member) uses it all the time more than anything else on our site
- it would be nice if it was faster
- likes icons in Vera
- likes that Vera is easy to find from home page
- one user was confused about what Vera was until she came to the library and got help using itOur interpretation: Glad this works.
- we plan to improve the "error/no results" screen (remind users to check for typos, etc)(See question # 5 above for other planned improvements to Vera).
As for the speed question, it's a known problem which we hope to address by perhaps upgrading to FileMaker 5 and considering other back end improvements. Some of the slowness can also be lessened by adjusting the number of results that are shown per screen for various types of searches -- something we plan to do in the near future.8. Find a page that allows you to request a book not owned by MIT.
(this question could have been worded more clearly- we had to explain that we meant borrowing, not purchasing)
No problem...
- except for one user who thought that "interlibrary borrowing" meant between libraries at MIT.
- another user thought the words "MIT only" that are under "book request" and "journal article request" on the page itself meant MIT materials only, instead of a page restricted to the MIT community.Our interpretation: We decided to add a sub-title to the page:
"Borrow materials not owned by MIT." And on the page itself we decided to change "MIT only" to "for MIT users only."9. Where would you go to find out if the Libraries own a particular videotape?
No problem.
- most users just assumed Barton would include videos
- a couple of users who weren't sure, successfully used the Search Our Collections chart to lead them to BartonOur interpretation: Glad this works.
10. How would you find out the phone number for the circulation desk at the Humanities library?
No problem
- user commented that it's good that it's so easy to find
- one user says there are too many links on the Humanities home page in generalOur interpretation: Glad this works.
We plan to test individual library home pages in a future test. For now the Humanities page has all of its subject guides listed, unlike Barker which links to another page that lists them all. Probably we should compromise by only showing a few sample subjects and then linking to the complete list on a separate page.
Post-test questions:
Our interpretation: All the usability studies say that it's not so important what people say about your site, it's what you see them doing when watching them find things. In spite of that, we were curious about people's likes and dislikes. Keep in mind that this is not a sample population (as in a formal survey), it's just a few comments by 8 of our users. We were quite happy to hear their comments.
1. Do you have any suggestions for improvements to the site?
- think of a better name for Interlibrary Borrowing
- make hours and locations one thing - put them together
- make Vera faster
- provide "jump to" letter within Vera subject results
- move "information access @mit" up next to "MIT libraries"
- get rid of icons on home page
- give detailed examples of boolean searching on our search engine page
- make virtual reference more obvious
- add a subject called "business & management"
- doesn't like "more..." links (another user really likes them)
- prefers old colors (another user prefers these new colors)
- wants a free-text search box instead of quick links
- doesn't like that colored columns are different widths
- make it easier to request older journal articles that aren't available electronically, perhaps links to ILB in Barton and Vera, or links to print holdings from VeraOur interpretation:
- add "looking for older back issues? check Barton" to results lists for e-journal searches in Vera
- (we've already addressed most of the others above)2. What do you like best about the site?
- likes the "more..." links on the home page ("doesn't box you in")
- likes that so many links are clickable on the home page
- likes these colors
- feels that the new site is faster
- likes the new category names on home page
- likes the photos
- likes the listing of links within each category on the home page
- likes the layout and design
- finds the site intuitive
- color bars help focus attention on functionality of the site ("that's why I would come here")
- much easier to find things (as opposed to the old site)
- likes the info for different audiences within quick links menu
- uses only Barton, Vera, and hours and finds these easy to find from home page
- likes the Search Our Collections chart - finds it helpful
- likes the About Us section
- likes the photos
- this site makes it clear what you can do on the site and what we have to offer (comment from a faculty member)Our interpretation:
We are very happy that people find it more useful. Doing the card-sorting usability tests really helped us to come up with categories that work. We were happy to hear the final comment above, made by a faculty member, since that was our goal.3. Which parts of the site are most useful to you?
- Search Our Collections and Subjects + Courses
- Barton, Vera, and our hours
- search functionality
- hours
- Barton, Vera, virtual reference and Ask Us
- Barton
- Search Our Collections4. About how often have you used the libraries¹ web site in the past?
___ never used (0)
___ only a few times (0)
___ a few times a year (1 user)
___ about once or twice a month (3 users)
___ every week (3 users)
___ every day (0)5. Is English your first language? If not, what is your first language?
- English - 5 people
- 1 Spanish
- 1 Turkish
- 1 forgot to ask
Future tests will focus on different audience groups: users with disabilites, faculty, alumni, administrative staff, outside users. We will also test more sections of our site. There are important pages that were left out of this test because they haven't been redesigned for the new site yet.