|
|
|
By Mark
Phillips, Head, Digital Project Unit, Information Technology
Services, UNT Libraries
O n November 22, 2005,
as was
reported in last month's Benchmarks Online, the Digital Projects Unit in the UNT Libraries carried out a harvest
of the "unt.edu" Web presence at the request of the University
Archivist. This harvest was executed using the Heritrix 1.4 Web
harvester built by the Internet Archive. The harvester ran for 14
hours before it hit the pre-determined limit of 1 million pages (which
was decided would be the maximum number of pages for this first
harvest of the UNT Web presence). If you noticed any problems that
were caused by the crawler on the day of the harvest, please send that
information to Mark Phillips (mphillips@library.unt.edu).
Currently the Digital Projects Unit is working on ways to provide
access to this data through browse and search functionality. If there
is anyone on campus who would like to use the harvested data for
research purposes, please contact Phillips to arrange for a
copy. Also, if your site was not included in this initial harvest and
you would like it to be included in the Spring
harvest scheduled for April 11th, send him the site URL.
Below is a breakdown
of some of the data that was harvested.
265 *.unt.edu hosts
found
[uri]
- (count)
ejde.math.unt.edu - (35386)
web2.unt.edu - (23890)
ecomod2.geog.unt.edu - (23537)
osprey.unt.edu - (23452)
rss.acs.unt.edu - (19459)
cops.csci.unt.edu - (19025)
www.hsc.unt.edu - (18765)
stoermer.coba.unt.edu - (15403)
esupport.csci.unt.edu - (15254)
acm.csci.unt.edu - (13980)
texinfo.library.unt.edu - (13503)
ieeecs.cse.unt.edu - (13296)
acm.cse.unt.edu - (13260)
students.csci.unt.edu - (12255)
www.unt.edu - (11543)
courses.unt.edu - (11246)
www.library.unt.edu - (11163)
govinfo.library.unt.edu - (10952)
callie.csci.unt.edu - (8719)
www.phys.unt.edu - (7602)
www.ias.unt.edu - (7558)
zope.unt.edu - (7493)
www.coe.unt.edu - (7367)
www.cas.unt.edu - (7344)
davinci.cascss.unt.edu - (6752)
www.cs.unt.edu - (6621)
humanitiespolicy.unt.edu - (5875)
www.biol.unt.edu - (5691)
www.hist.unt.edu - (5434)
orgs.unt.edu - (5188)
www.chem.unt.edu - (4724)
www.psci.unt.edu - (4447)
www.engl.unt.edu - (4047)
www.comm.unt.edu - (3752)
www.coba.unt.edu - (3466)
www.geog.unt.edu - (3420)
www.math.unt.edu - (3239)
www.tcet.unt.edu - (2894)
kallmanreport.math.unt.edu - (2511)
www.etec.unt.edu - (2419)
www.psyc.unt.edu - (2223)
webctvista.unt.edu - (2153)
dev.coba.unt.edu - (2056)
www.tams.unt.edu - (2010)
logic.csci.unt.edu - (1971)
www.phil.unt.edu - (1934)
www.kntu.unt.edu - (1908)
www.rtvf.unt.edu - (1742)
www.cep.unt.edu - (1707)
courseweb.tac.unt.edu - (1638)
notices.infosec.unt.edu - (1506)
mayborninstitute.unt.edu - (1504)
alt.coba.unt.edu - (1394)
listserv.hsc.unt.edu - (1300)
people.unt.edu - (1288)
irservices.library.unt.edu - (1159)
digitalprojects.library.unt.edu - (1139)
www.art.unt.edu - (1134)
www.music.unt.edu - (1127)
essc.unt.edu - (1121)
kevski.cecs.unt.edu - (1078)
www.ecoplex.unt.edu - (1037)
www.jour.unt.edu - (1026)
web2survey.unt.edu - (943)
inhouse.unt.edu - (914)
www.iittl.unt.edu - (862)
www.ling.unt.edu - (795)
www.solar.unt.edu - (724)
www.mtsc.unt.edu - (702)
www.international.unt.edu - (628)
courseweb.unt.edu - (625)
libwebtest.library.unt.edu - (606)
www.wmst.unt.edu - (576)
www.cse.unt.edu - (565)
www.cps.unt.edu - (514)
www.eng.unt.edu - (480)
docs.cecs.unt.edu - (437)
web2caari.unt.edu - (412)
www.econ.unt.edu - (410)
www.gal.unt.edu - (397)
www.forl.unt.edu - (356)
www.smhm.unt.edu - (349)
www.philtest.unt.edu - (295)
fermat.csci.unt.edu - (281)
www.attd.unt.edu - (280)
www.efec.unt.edu - (275)
www.workteams.unt.edu - (268)
projectweb.cc.unt.edu - (253)
ncs.unt.edu - (234)
students.hsc.unt.edu - (234)
poseidon.csci.unt.edu - (210)
secure.cep.unt.edu - (190)
skytheater.unt.edu - (182)
library.hsc.unt.edu - (180)
www.sphs.unt.edu - (173)
emod.unt.edu - (168)
forums.unt.edu - (164)
larc.csci.unt.edu - (157)
www.opgf.unt.edu - (155)
gwe.hsc.unt.edu - (150)
research.hsc.unt.edu - (145)
www.inst.unt.edu - (145)
www.slis.unt.edu - (143)
www.peace.unt.edu - (137)
www.paths.unt.edu - (121)
classes.cecs.unt.edu - (119)
gwf.hsc.unt.edu - (117)
thecampsite.ec.unt.edu - (114)
prime.cecs.unt.edu - (107)
www.sova.unt.edu - (107)
noc2.datacomm.unt.edu - (103)
www.mms.unt.edu - (103)
www.csci.unt.edu - (100)
www.ee.unt.edu - (97)
library3.hsc.unt.edu - (96)
max.lis.unt.edu - (93)
www.debate.unt.edu - (92)
careercenter.unt.edu - (91)
www.pacs.unt.edu - (89)
webcpp.csci.unt.edu - (88)
eiswbdm.admin.unt.edu - (83)
zeus.csci.unt.edu - (82)
www.tsgs.unt.edu - (76)
eiswbtrul.admin.unt.edu - (73)
webmail.csci.unt.edu - (71)
www.cmp.unt.edu - (69)
www.gsc.unt.edu - (64)
www.edsp.unt.edu - (63)
asiaalacarte.ec.unt.edu - (62)
xtremetravel.ec.unt.edu - (62)
eiswbdv.admin.unt.edu - (60)
www.healthcenter.unt.edu - (59)
hera.csci.unt.edu - (59)
eiswbqa.admin.unt.edu - (58)
mediabook.library.unt.edu - (58)
itds2.unt.edu - (56)
www.facilities.unt.edu - (54)
www.plant.unt.edu - (53)
endow.unt.edu - (53)
tla.library.unt.edu - (51)
cerl.unt.edu - (51)
www.cecs.unt.edu - (50)
www.danceandtheatre.unt.edu - (50)
www.year2000.unt.edu - (50)
dining.unt.edu - (50)
untsystem.unt.edu - (49)
home.unt.edu - (48)
hammer.lis.unt.edu - (47)
search.library.unt.edu - (45)
newdirections.unt.edu - (45)
leonardo.cascss.unt.edu - (42)
www.sportpsych.unt.edu - (39)
scubasafari.ec.unt.edu - (37)
cri.hsc.unt.edu - (35)
etravel.ec.unt.edu - (34)
nervous.cnns.unt.edu - (33)
webct.unt.edu - (32)
ciso.unt.edu - (32)
mrtg.datacomm.unt.edu - (32)
elm.cecs.unt.edu - (31)
lit.csci.unt.edu - (31)
dataweb.cecs.unt.edu - (30)
goliath.music.unt.edu - (30)
ostmed.hsc.unt.edu - (28)
lug.unt.edu - (28)
mira.csci.unt.edu - (27)
roanoke.ec.unt.edu - (27)
www.crp.unt.edu - (26)
www.csam.unt.edu - (26)
texasopti.hsc.unt.edu - (25)
jsp.csci.unt.edu - (24)
my.unt.edu - (23)
untvn.datacomm.unt.edu - (23)
www.murphycenter.unt.edu - (23)
cgi.cascss.unt.edu - (23)
www.physics.unt.edu - (22)
www.cascss.unt.edu - (21)
wcherry.math.unt.edu - (21)
netstorage.coba.unt.edu - (21)
thptest.library.unt.edu - (20)
www.forensic.unt.edu - (20)
data3.unt.edu - (19)
efnhmuseum.unt.edu - (19)
eiswblspb.admin.unt.edu - (19)
www.nrl.csci.unt.edu - (19)
www.tcee.unt.edu - (16)
smtmcg.acs.unt.edu - (15)
www.isq.unt.edu - (15)
www.logistics.unt.edu - (15)
www.csrl.unt.edu - (14)
nds09.unt.edu - (14)
nds08.unt.edu - (14)
www.mcdu.unt.edu - (13)
www.water.unt.edu - (13)
www.texasbusmark.unt.edu - (12)
eiswbcp.admin.unt.edu - (12)
www.elmfork.unt.edu - (12)
aquaticecology.unt.edu - (12)
simeon.acs.unt.edu - (12)
nsl.unt.edu - (11)
www.css.unt.edu - (11)
eiswbtra.admin.unt.edu - (10)
untss.unt.edu - (10)
digitallab.lis.unt.edu - (10)
dosaf.hsc.unt.edu - (10)
eaglemail.unt.edu - (10)
www.cics.unt.edu - (10)
counseling.unt.edu - (9)
pcb2.acs.unt.edu - (9)
libwsec.library.unt.edu - (8)
e-vents.ec.unt.edu - (8)
www.inhouse.unt.edu - (8)
advantage.ec.unt.edu - (7)
realestatealumni.unt.edu - (7)
www.ttis.unt.edu - (7)
survey.unt.edu - (7)
nss.ec.unt.edu - (7)
media.unt.edu - (6)
csrl.csci.unt.edu - (6)
gwwac.unt.edu - (6)
checkin.acs.unt.edu - (6)
www.vdcl.cse.unt.edu - (6)
myls.unt.edu - (6)
eiswbcv.admin.unt.edu - (5)
myfs.unt.edu - (5)
zanovar.csci.unt.edu - (5)
bcisnt1.coba.unt.edu - (4)
housing.unt.edu - (4)
www2.hsc.unt.edu - (4)
www.seeq.com - (3)
surveys.coba.unt.edu - (3)
www.securityid.unt.edu - (3)
bulkmail.unt.edu - (3)
untc2.unt.edu - (3)
slis-106254.lis.unt.edu - (3)
js2.ec.unt.edu - (3)
info.unt.edu - (3)
quadpro.ec.unt.edu - (3)
pan.acs.unt.edu - (3)
training.unt.edu - (2)
www.epsy.unt.edu - (2)
my.hsc.unt.edu - (2)
basicgroupwise.unt.edu - (2)
metamarc.lis.unt.edu - (2)
ba235e.coba.unt.edu - (2)
cbt.acs.unt.edu - (2)
rcs.cse.unt.edu - (2)
www.communications.unt.edu - (2)
tsmt.unt.edu - (2)
iii.library.unt.edu - (2)
webasm.csci.unt.edu - (2)
eiswbeppb.admin.unt.edu - (1)
gww.unt.edu - (1)
eiswbfspb.admin.unt.edu - (1)
libharvest.library.unt.edu - (1)
purchase.tickets.com - (1)
eiswbpapb.admin.unt.edu - (1)
murphycenter.unt.edu - (1)
hsclibrary.hsc.unt.edu - (1)
www.datacomm.unt.edu - (1)
unt90841.cecs.unt.edu - (1)
gwim.unt.edu - (1)
gwweb.hsc.unt.edu - (1)
www.tac.unt.edu - (1)
dls.unt.edu - (1)
Mimetype harvested:
125 mimetypes
harvested
[mimetype]
- (count)
text/html - (361342)
image/gif - (158470)
image/jpeg - (110557)
application/pdf - (29801)
text/plain - (24049)
application/msword - (7772)
application/vnd.ms-powerpoint - (3950)
image/png - (3038)
text/htm - (2884)
text/css - (2475)
application/x-httpd-php - (2277)
application/x-dvi - (1807)
application/x-javascript - (1753)
application/octet-stream - (1618)
application/xml - (1504)
application/postscript - (1218)
text/xml - (1177)
application/vnd.ms-excel - (1124)
application/x-gzip - (1107)
java/* - (954)
application/x-shockwave-flash - (823)
audio/x-pn-realaudio - (625)
application/zip - (529)
audio/mpeg - (394)
audio/x-wav - (379)
text/x-java - (364)
text/x-c+src - (333)
image/x-icon - (320)
application/x-perl - (317)
application/x-java-vm - (259)
video/quicktime - (246)
text/x-perl - (190)
video/mpeg - (185)
video/x-msvideo - (183)
text/x-chdr - (180)
image/bmp - (178)
application/x-zip-compressed - (178)
application/x-tex - (156)
text/rtf - (125)
chemical/x-pdb - (125)
application/x-tar - (121)
application/x-rpm - (104)
text/x-tex - (97)
image/x-eps - (94)
audio/midi - (93)
application/powerpoint - (80)
no-type - (80)
x-world/x-vrml - (79)
application/x-msdos-program - (72)
video/unknown - (67)
application/rtf - (66)
audio/basic - (63)
application/x-java-archive - (46)
application/vnd.rn-realmedia - (44)
text/javascript - (44)
application/x-msdownload - (43)
image/tiff - (42)
image/pjpeg - (40)
text/x-csrc - (37)
application/x-msmetafile - (29)
application/x-ms-wmz - (22)
audio/wav - (21)
text/x-python - (20)
image/x-ms-bmp - (19)
video/x-ms-asx - (18)
application/x-troff - (14)
application/x-object - (12)
application/x-stuffit - (11)
application/x-executable - (11)
audio/x-realaudio - (10)
application/x-ica - (10)
text/x-c src - (9)
application/x-trash - (8)
video/x-ms-asf - (7)
image/gi - (7)
video/x-ms-wmv - (7)
application/atom+xml - (7)
message/rfc822 - (5)
application/x-xfig - (4)
text/html - (4)
application/x-java-jnlp-file - (4)
audio/x-mpegurl - (4)
application/smil - (4)
application/java - (4)
application/x-mspublisher - (4)
text/x-component - (3)
audio/unknown - (3)
application/rdf+xml - (3)
application/mac-binhex40 - (3)
image/x-xbitmap - (2)
text/comma-separated-values - (2)
audio/x-aiff - (2)
application/vnd.sun.xml.writer - (2)
application/x-pointplus - (2)
application/excel - (2)
application/x-java-applet - (2)
text/x-c++hdr - (2)
text/plai - (2)
application/x-debian-package - (2)
application/xhtml+xml - (2)
text/vnd.wap.wml - (2)
audio/mid - (1)
text/calendar - (1)
application/x-apple-diskimage - (1)
message/news - (1)
application/x-redhat-package-man - (1)
application/x-shellscript - (1)
application/x-msi - (1)
chemical/x-xyz - (1)
text/vbscript - (1)
application/octetstream - (1)
application/x-shockwave-flash2-p - (1)
application/java-vm - (1)
application/java-archive - (1)
application/rss+xml - (1)
application/x-director - (1)
application/vnd.ms-project - (1)
text/js - (1)
application/audio/x-ms-wma - (1)
application/winhlp - (1)
application/vnd.visio - (1)
application/xml-dtd - (1)
text/x-vcard - (1)
image/vnd.microsoft.icon - (1)
Return to top |
