"Big Test" usability test results

Quantitative results
and answers to pre- and post-test questions

Test A
(4 undergrads, 4 grads)

Department of each student:
STS
STS
comparative media studies
physics
media lab
Mech E
CS
unknown

Pretest questions:
1. Have you used the MIT Libraries’ web site before? If so, about how often?
(5 yes, 1 no)
- Yes, at least once a week.  He uses ILB/RSC a lot.
- Yes, about once a day.
- Yes - at least every week, maybe more often.
- Yes. Checks it 2-3 times per semester for hours, etc.
- Yes, daily.
- No.

2. Have you had a library instruction class?
(2 yes, 5 no)
- No.
- No.
- Yes, orientation class.
- No, but worked in library during junior high school.
- No, but worked in library during high school.
- No.

3. Have you ever asked for help from a librarian either in person, by phone,
email or live chat?
(6 yes)
- Yes, in person.
- Yes, in person.
- Yes, when looking for a film that was hard to find in Barton -- asked
librarian at orientation.
- Yes, in Hayden.  Wanted help with bound journals, and staff person took
him right to the basement and showed him around.  Very pleased with service.
- Yes, has used Ask Us email, and Ask Us Live.
- Yes, in person, a few times (this was the circ/information desk, not reference)

 

Test A, searching for known items:
Download test A - PDF

(A "yes" indicates that they completed the task successfully).

1. can they find article that we have in print only, through Barton?
do they use Barton or Vera first?
can they find a way to have it delivered electronically from Doc Services?
yes - tried Vera, BELL, Barton, in that order
yes - Barton
yes - Barton
yes - Barton
yes - Barton
yes - Barton

2. can they find an article that we have online and in print?
do they use Barton or Vera first?
yes - Vera
yes - Barton
yes - Barton, needed help to find it, though
yes - Barton
yes - Barton, but didn't find e-version, only print
yes - Barton, but thought we had it in print only

3. article not available in print or online at MIT
do they use Barton or Vera first?
can they find ILB journal request form?
yes - tried Vera, then BELL, then Barton, then ILB
yes - tried Barton, then Vera, then BELL, then ILB
yes - tried Barton, then HOLLIS, then ILB
yes - tried Barton, then ILB
yes - tried Barton, then ILB
yes - tried Barton, then ILB

4. book on shelf in Dewey, can they find whether it's checked out?
can they find BookPage to have it sent to a closer library?
yes, except BookPage, yes, yes, yes, yes, yes, except BookPage

5. book in RSC, can they find RSC request form to get it delivered?
yes, yes, yes, yes, yes, yes

6. can they find our patents guide?
yes, yes, yes, yes, yes, yes

7. can they find our topographic maps guide?
yes, yes, yes, yes, yes, no

8. home page: where would you go for help?
ask us link/phone
Info Navigator
research help/email
research help/phone
vera subjects/ask us
ask us/live chat

Post-test questions:

What improvements to our site would help you the most?
1. link to Vera from Barton
list of what he *can* do from the web page (map info, patent info, etc.) 
[how do you know what's buried a few layers down?  how to even know
to look for it if you don't know we have it?]
separate box on the front page for types
Comment: sometimes when you're in Barton the back button won't let
you go back... it won't let you get out of Barton.

2. She looks for journals in Barton, Vera and Bell... it would be helpful to
have a way to search all of them at once.
Comments: the "website is excellent and is by far the best library website"
she's used
Because she's doing history, she often gets ILB, books from Harvard, etc.
and uses JSTOR a lot.

3. column menu is not in the catalog.
linking request forms to the Barton record so you don't have to cut and paste.
search engine to cross-search books, journals, etc. all together -- as an
option (would be helpful for incomplete references).
DVD, CD searching not so buried.

4. -None.  "The Ask Us thing was kind of hard to use, but that's someone
else's software."  He really likes the library, and thinks the site is very good.

5. Cannot get Your Bookshelf to maintain data about herself from session
to session (same complaint as with ILB forms.  Is this mysterious?  Is
there a problem with her machine?)
Reiterated idea that ability to place holds on "In Library" items would
be very helpful.
Likes the libraries, and the online services, very much.  Appreciates the
databases, catalog, resources, and staff.

6. User said it would be good if it were more apparent right on the home
page that you can get articles online.  Also the home page should show
a list of databases.  Also the home page should show that you can have
articles shipped (I asked for clarification on this, and he seemed to mean
ILB).  Essentially, the tester wants everything on the home page.

-----------------------------------------------------------------------------------------
Test B
(5 undergrads, 8 grads)

Department of each student:
Mech E
managerial sciences
chemical engineering
EAPS
materials science
cognitive science
biology
biology
EECS
STS
nuclear engineering
EECS
unknown

1.  Have you used the MIT web site before?
(10 yes)
- Yes, about 3 times per week
- Yes, about half a dozen times this semester
- Yes, uses it a lot.  Every day.
- Yes, uses MIT Libraries web site every couple of weeks.
- Yes, uses MIT Libraries web site 1-2 times a week.
- Yes, uses MIT Libraries web site every day.
- Yes, uses MIT Libraries web site about once a week.
- Yes. A few times a month.
- Yes, uses MIT Libraries web site about once a week.
- Yes, multiple times daily.

2.  Have you had library instruction class?
(3 yes, 7 no)
- No.
- Yes, maybe, a long time ago, in Schering Plough.
- No.
- Yes, had one general instruction class in Lindgren.
- No, has not taken any library instruction classes at MIT (has at other
schools).
- No, has not taken any library instruction classes at MIT.
- No, has not taken any library instruction classes at MIT.
- No.
- Yes, had some instruction while taking 9.00.
- No.

3.  Have you asked for help from a librarian?
(9 yes, 1 no)
- Yes, in person.
- Yes, went to the reference desk at Dewey, they were "pretty helpful".
- Yes, asked for help in person.
- Yes, has emailed a librarian for help 1-2 times in the past.
- No, has not asked a librarian for help at MIT.
- Yes, has asked a librarian for help in person about ILB and finding
books on the shelf.
- Yes, has asked a librarian for help in person and by phone.
- Yes. In Person.
- Yes, has talked to a librarian and used ILB.
- Yes. All of the above, except for chat. Usually by phone.

Test B, open-ended searching
Download test B - PDF

(A "yes" indicates that they completed the task successfully).

1. Vera, find 2 databases:
yes, yes, yes, yes, yes, yes, yes, yes, yes, no

2. Search a database, find 2 articles:
yes, yes, yes, yes, yes, yes, yes, yes, yes, yes

3. Barton, find 2 books:
yes, yes, yes, yes, except Your Bookshelf, yes, yes, yes, yes,
yes except Availability link, yes

4. SFX - find out if we have full text:
(Some people clicked on link to publisher's web site first, some people
clicked on "libraries that own this item" first, but they all eventually
found the SFX link. They all used the text link instead of the button).
link - yes, menu - yes
link - yes on 2nd try, menu - yes
link - yes on 2nd try, didn't use menu for ILB
link-yes, menu-yes
link - yes on 2nd try, menu - yes
link - no, menu - yes
link - yes, menu - yes
link - no, menu - yes
link - yes on 2nd try, didn't use menu for ILB
link - yes on 2nd try, menu - yes

5. Subject Guides - find 2 resources
not sure, somewhat, no, yes, yes, yes, yes, yes, no, sort of
(several people had trouble noticing sidebar)

6. home page: where would you go for help?
what they chose first:
ask us link/email help via web form
science lib home page - phone # of ref desk
research help/help yourself
ask us link/email help via web form
ask us link/live chat
help yourself
About Us/Contact Us
research help/email help via web form
research help/ask us
ask us link/live chat

Post-test questions:

What improvements to our site would help you the most?

1.
* User assumes that if she can't find what she needs, the library
doesn't have it. 
* When looking for journals, she looks at the article titles for information
on the topic of the article, but that is not always clear. (I showed her
the database descriptions in Vera, which can help her decide what database to use).
* she mentioned wanting to be able to see what items she has checked
out online (and other Your Account features).  I explained Your Account to her.
* one thing I noticed about her searching.  She tended to use the
keyboard instead of the mouse if possible.  Like using the page down
key instead of scrolling with the mouse.
* she wasn't a big talker, despite my promptings.

2.
*"in Barton, if you mark a record, it automatically becomes the first record
in the list and changes the order if you have many records. Is there a way
to change that?"
*would like to be able to download records directly into EndNote.
*would like it if old materials in the RSC could be searched in Barton.
*would like to be able to search USGS maps in Barton more easily.
*would like more information on how to use databases, but not as a class.
(I showed him the database cheatsheets, as well as the EndNote
cheatsheet and the Maps publication type guide. He was unaware of all three.)

3.
*showed her the side bar on the subject pages.  She ignores side bars
because they usually aren't useful, especially on library pages
* showed her SFX page and the options there
*When she goes to Barton from the home page, then wants to go back to
the home page, if she uses the browsers back button it keeps her in Barton. 
Then she has to use other browser features to get her back.  That is annoying.
* descriptions of what you can find in each database should be better (I
showed her the database manager and spreadsheet that Maggie created
at Dewey and she liked them).

4.
*showed him the Chem Eng/Chem subject page and his subject specialist
contact info.  Suggested he make an appointment and he was eager to do that.
* he borrows a lot of books that are due at different times/dates.  He
would like to be able to look at his records online.  He had this capability
at his previous university.  He doesn't want to have to "bother" the circ
staff by calling them. (I told him about Your Account and how we hoped
to have that ready next semester).

5. Barton options confusing, as mentioned in #3.
Wants a telnet version of Barton back.

6. Her old school had a printed list of database descriptions organized by
subject that worked well for her. Thought this could be online too - I wish
I had asked what was missing with Vera in this need.

7.
- In Barton: would really like to have more than 10 results per screen,
would like to have 50 per screen, or as many as possible.
- In Vera: Would like to have an advanced search screen that let you limit
to peer-reviewed journals vs. general review journals, would like to have
more options for helping him choose the best databases, would like an
advanced search that let you search the descriptions as well as the titles
- would like some place to store his personal list of favorite databases
and e-journals and books

8.
- In Barton: would like the various navigation links grouped more logically
(search screens vs. custom features), suggests a different color for the
custom features to make them stand out more. She likes those features,
but didn't notice we had them.
- In subject guides: feels we should focus on links to web sites and links
to labs and departments at MIT (things NOT in Barton) and forget about
lists of books or reference books - she would use Barton to find those.

9.
*would like to have more book descriptions in Barton. Some books have
descriptions (abstracts) others don't.
*wasn't familiar with using First Search and it was frustrating to search
the protocols database.
(FirstSearch timed out while we were looking at the subject pages and
she kept getting access denied messages because a specific protocol
database selection is required, and protocol db's that we aren't
subscribed to show up.)
I showed her the database cheatsheet for FirstSearch.

10.
- Make hours more obvious
- Icons were artistically interesting, but didn't know what they were or
what the point was.  Icons should have alt tags saying what they would go to.