Web Guidelines:
MIT Libraries
- Accessibility
- Barton links
- Best practices for web interfaces of searchable databases
- Blog: library news
- Cascading style sheets
- Directory structure
- Dreamweaver at MIT
- Dome logo
- Fonts & text colors
- Forms
- FTP at MIT
- Google Analytics
- Graphics
- Guidelines for MIT Web Sites
- Hiding pages from search engines
- Javascript
- LibGuides
- Link-checking
- Links to a web page
- Media: animation, etc.
- meebo
- News blog
- PDFs
- Principles of our web site
- Publishing checklist
- Restricting pages to MIT only
- Server-side includes
- Shortcut URLs
- Staff Photos
- Statistics
- Subject pages guidelines
- Subject selector links
- Templates
- Title tags
- URLs & file names
- Video tutorials
- Writing
- Wikis
[Public
web]
[Staff web]
Web Guidelines: MIT Libraries
Hiding pages from the search engines
Method 1 - add a "no robots" meta-tag
Use this method to temporarily hide an individual page from the search engine. Add this meta-tag to the head section of the page:
<head> <meta name="robots" content="noindex,nofollow"> </head>
The disadvantage of this method is that you have to remember to remove the tag when you are ready to make the page live and want the search engine to find it.
Method 2 - use a directory that is hidden from the search engine
The web manager can tell the search engine which directories of our site not to index by means of a "robots.txt" file at the root level of our site.
You may view our "robots.txt" file here: http://libraries.mit.edu/robots.txt
Each library has a test directory: "dewey-test", for example. You may use these directories as a place to test pages in progress. They are excluded from the search engine at MIT. There is also a "guides-test" directory for guides that are in progress.
If you are creating a new directory that you want hidden from the search engine, contact the web manager (hennig@mit.edu) and ask to have it added to the "robots.txt" file (before you create it).