Digital Library of MIT Theses usability test results
Site tested: http://theses.mit.edu
Christine Quirion, Technical Supervisor, Document Services
June 2003The Test
The Digital Library of MIT Theses (DLMITT) Usability Test took place between April and May of 2003. 11 users participated, including graduate students, library staff, and administrative staff from academic departments. Users were given 5 tasks to complete using the DLMITT, asked post-test questions, and asked to do the same with an alternate image display system at another institution.Observers: Margret Branschofsky, Katherine McNeill-Harman, Christine Quirion, Tim Sheehan, Sarah Williams. Nicole Hennig assisted by providing feeback in the creation of the test, preparing observers, and brainstorming solutions.
The following is a draft of the observed problems and brainstormed solutions from a meeting on June 5, 2003. The solutions will be evaluated for future implementation, and considered before the DLMITT collection is transferred to MIT's Dspace repository.
Quantitative Results:
- Users were able to accomplish the desired tasks 77% of the time.
- 3 users answered all five questions correctly.
- 4 users answered four of five questions correctly.
- 3 users answered three of five questions correctly.
- 1 user answered two of five questions correctly.
- 7 users could not answer question 5 correctly.
Changes that could be easy to implement:
- change terminology: organizations to collections?
- altering page footer to be more like a navigation bar (options to Basic Search, Advanced Search, Price Info, Help, Document Services), try to add same bar to top of page if possible
- on search results screen, list e-theses at beginning of results list
- modify and simplify order form by adding bullets to top and highlighting free option to MIT users
- add links to Document Services and contact information to link users to help
- edit Search Tips page, model after and FAQ, clarify Barton searching options
Changes that may be difficult to implement:
- Create consistent look and feel among pages by adding headers and navigation bars
- Change Document Summary screen to highlight thumbnail viewing option by changing wording on the options, eliminate inline gif drop-down menu, change names of links to be more descriptive
- Change terminology: organizations to collections?
- Search Results Screen: merge both e-thesis and scanned thesis results, consider removing the system numbers at the end of each thesis, display degree year instead
- Thumbnail image display: make navigation options more prominent
- Change links to Search from lower level pages to point back to home page/simple search
- move Copyright Notice on Document Summary screens
Changes Requested by users for long term:
- ability to download PDF file directly
- links directly to online theses from Barton
- searching by department, subject, thesis supervisor, as well as advanced searching options
- more theses, full text
- sorting of search results by date, author name, title words, relevance, highlighting search terms within results set
- linked table of contents to skip to chapters and sections of theses
Thumbnail Images
- The eleven users tested were all members of the MIT community. MIT users were all interested in getting the PDF version of the document quickly, preferably via self-service downloading, as is offered in other electronic resources.
- Note: Requests made to the etheses-admin@mit.edu mail list indicate that outside users make a heavy use of the thumbnail images, and purchase copies of the documents on a regular basis.
- Note: Were the volunteer observers to administer this test again, users would be forced to respond with a numerical indication of the importance of thumbnails. Hindsight revealed that this question may not have been worded appropriately to elicit a specific response.
- Of the eleven users tested, the responses to post-test question number 4 were as follows:
- 1 user indicated that the thumbnails were an essential feature
- 5 users indicated that the thumbnails are useful for viewing specific pages, charts, figures, or image material
- 1 user felt the thumbnails were not essential, and in fact difficult to use
- 4 users did not give a specific response to this question
- Of the eleven users tested, three did not see the link to"view thumbnail pages" at all.
Usability Issues by Category [Download Test in PDF format]
Advanced search: users confused by "select one or more organizations from this list" -change terminology- all theses in this collection are electronic even if they were scanned.
-substitute "collections" for "organizations"
Users had a hard time getting back to home from advanced search -consider adding navigation bar to page, or adding a header that makes the pages consistent with the home page Users confused by advanced search. not clear how it is different from simple search users would like subject and department searching, advanced search option is available in Dspace Search Results [view Search Results Screen]
search results page too busy, users confused by the "organizations" and "experimental thesis" terminology -consider consolidating search results together, offer new etheses results at the top of result list
-look for other method to delineate electronically submitted theses in the individual thesis record, rather than in results display
Search results display unclear and hard to read -put actual search terms used up at the top of the page or off to the side (like Google) so they take up less space
-put collection info (organizations) off to the side, or eliminate
-users would like to be able to sort results or see search terms highlighted in result screen (KWIC)
Users confused by thesis numbers (2002-212) -consider showing the date of the thesis, but dropping the number after, or putting it elsewhere and calling it a system number.
-users asked if the second number referred to the number of pages, or a date?
When two theses had the same year number, users were unclear which one was most recent -see above Users missed e-theses in search results because they appear at the bottom of the screen, below the fold -place results in reverse-chronological order, with e-theses at the top Users did not see or understand "link to thumbnail pages" -change link text to read: "preview document," create an icon that shows a small version of the thumbnail pages Users did not notice "display selected page" option, confused by "inline gif image" -remove drop down menu that says "inline gif," change wording to "go to page [box to enter page number][go button]" In some theses, page numbers and thumbnail numbers do not correspond -not sure this can be improved without additional metadata User that went into RFC bib record could not get back to search page, was not sure what they were looking at -add footer/nav bar to all pages, make links to Home more prominent
-link to "Search" above footer links to Advanced search, not to basic search/home
-consider having same page template and nav bar for all pages, including deeper pages like bib info
-consider using description other than "Bibliographic Information RFC..."
Users weren't able to find out length of document without paging through thumbnails -include number of pages with description of document, or in search results list
User zoomed in on a page accidentally, not sure how to get back into document -consider adding a navigation bar/modifying footer to link users back to the document summary, thumbnails, basic search, advanced search, help, document ordering info User had trouble with thumbnail navigation -user "got tired" of clicking "next page" to get between pages, and between sections
-consider making navigation more prominent at top and bottom, or using icons to illustrate?
Users didn't see link to order PDF or bound paper version -consider changing the wording- "request a PDF or paper copy of this thesis", have link to Pricing Information next to it Users not sure where to go for help if they can't find a thesis or are confused about ordering -did not pick "Ask Document Services" link from home page footer because he just didn't see it
-did not pick "Comments/Questions" form because she thought it was for comments on the site, not a place where someone would respond with help
User not sure how to get back to simple search from within a document -addition of navigation bar to pages could help, or making link back to Home and Search more prominent Links to Search from thumbnail pages go back to Advanced Search not basic search -consider linking users back to basic search first, or offering two links, one to basic search, one to advanced Search Tips page refers users to Barton if they can't find theses. Users have trouble getting back to DLMIT. Once in Barton, users think they will be taken from Barton to the theses they want.
-modify Search Tips to indicate that searching Barton will allow you to find paper copies of ALL MIT Theses
-consider linking to theses directly from Barton via SFX (Dspace would accomplish this) or adding links to the thesis records in Barton
-scope of collection confusing to users, clarify in Search Tips
Users thought they had to purchase the thesis to print it, and couldn't find price options -make option to view page images more prominent,change link text to read: "preview document,"as above
-adjust order form- add a few bullets above form with information from ordering theses page, or link to Ordering Theses (/docs/theses.html) instead of linking to the form directly
User not clear on how to get entire document in PDF format. MIT users weren't aware of free PDF option or MIT pricing -consider making changes to order form, add bullets to top as noted above
-Allowing self-service PDF downloading option for MIT users in Dspace would resolve this problem.
Users confused by order form -the first thing listed on the form is credit card related info. need a prominent link to free versions for MIT users, similar to text in docs/theses.html
-consider offering a form just for thesis requests to limit possible choices and make it simpler for users
-User confused by browser security warning after clicking link to order form. Consider changing link text to make it clearer: "order this thesis in PDF or paper format online via secure web order form?"
Copyright notice and placement is confusing -notice takes up a lot of space, and has a great deal of prominence due to it's location. consider changing the order it appears, or placing to one side?
-this notice provides the only link to Document Services via email on the results pages. the contact info should be higher up on the page.
Usability Test Responses for each Question:
users did not see or understand "link to thumbnail pages" -change link text to read: "preview document," create an icon that shows a small version of the thumbnail pages users thought they had to purchase the thesis to print it, and couldn't find price options -make option to view page images more prominent as above
-adjust order form- add a few bullets above form with information from ordering theses page, or link to ordering theses instead of linking to the form directly
users did not notice "display selected page" option, confused by "inline gif image" -remove drop down menu that says "inline gif," change wording to "go to page [box][go button]" search results page too busy, users confused by the "organizations" and "experimental thesis" terminology -consider consolidating search results together, offer new etheses results at the top of result list
-look for other method to delineate electronically submitted theses in the individual thesis record, rather than in results display
in some theses, page numbers and thumbnail numbers do not correspond -not sure this can be improved without additional metadata users confused by advanced search. not clear how it is different from simple search users would like subject and department searching, advanced search option is available in Dspace user zoomed in on a page accidentally, not sure how to get back into document -consider adding a navigation bar/modifying footer to link users back to the document summary, thumbnails, basic search, advanced search, help, document ordering info Question 2:
- user not sure how to get back to simple search from within a document -see above -advanced search: users confused by "select one or more organizations from this list" -change terminology- all theses in this collection are electronic even if they were scanned.
-substitute "collections" for "organizations"
-Search Tips page refers users to Barton if they can't find theses. Users have trouble getting back to DLMIT. Once in Barton, users think they will be taken from Barton to the theses they want.
-modify Search Tips to indicate that searching Barton will allow you to find paper copies of ALL MIT Theses
-consider linking to theses directly from Barton via SFX (Dspace would accomplish this) or links in the thesis records in Barton
-scope of collection confusing to users, make this more prominent in Search Tips
-user that went into RFC bib record could not get back to search page -add footer/nav bar to all pages, make links to Home more prominent
-link to "Search" above footer links to Advanced search, not to basic search/home
-consider having same page template and nav bar for all pages, including deeper pages like bib info
-users weren't able to find out length of document without paging through thumbnails -include number of pages with description of document -search results display unclear and hard to read -put actual search terms used up at the top of the page or off to the side (like Google)
-put collection info (organizations) off to the side
-users would like to be able to sort results or see search terms highlighted in result screen
-users missed e-theses in search results because they appear at the bottom of the screen, below the fold -place results in reverse-chronological order, with e-theses at the top -users confused by thesis numbers (2002-212) -consider showing the date of the thesis, but dropping the number after, or putting it elsewhere and calling it a system number.
-users asked if the second number referred to the number of pages, or a date?
-when two theses had the same year number, users were unclear which one was most recent -see above -user had trouble with thumbnail navigation -user "got tired" of clicking "next page" to get between pages, and between sections
-consider making navigation more prominent at top and bottom, or using icons to illustrate?
-user not clear on how to get entire document in PDF format. MIT users not clear on if it is free -consider making changes to order form as noted above -users not clear on pricing options for MIT -consider making changes to order form as noted above. Self downloading option for MIT users in Dspace would resolve this problem. -users didn't see link to order PDF or bound paper version -consider changing the wording- "request a PDF or paper copy of this thesis", have link to Pricing Information next to it -user confused by browser security warning after clicking link to order form -change link text to make it clearer: "order this thesis in PDF or paper format online via secure web order form?" -users confused by form -the first thing listed on the form is credit card related info. need a prominent link to free versions for MIT users, similar to text in docs/theses.html
-consider offering a form just for thesis requests to limit possible choices and make it simpler for users
-users not sure where to go for help if they can't find a thesis or are confused about ordering -did not pick "Ask Document Services" link from home page footer because he just didn't see it
-did not pick "Comments/Questions" form because she thought it was for comments on the site, not a place where someone would respond with help
-copyright notice is confusing -notice takes up a lot of space, and has a great deal of prominence due to it's location. consider changing the order it appears, or placing to one side?
-this notice provides the only link to Document Services via email on the results pages. the contact info should be higher up on the page.
