Features you should be aware of

• It needs to find links somewhere

• It ignores duplicates

• It ignores URLs with query arguments

• It only indexes http: (not https:, ftp:, gopher:)

  • and ignores JavaScript, Flash, etc

• It can't get at passworded resources

• It can have problems with what look like 'huge' sites

• It is 'inside' the University, but externally accessible