Access the full text.
Sign up today, get DeepDyve free for 14 days.
Loading next page...
/lp/sage/a-web-crawler-design-for-data-mining-rXjK3C0AIu
References (31)
-
M. Thelwall (2001)
Commercial Web site linksInternet Res., 11
-
(1997)
Structuring and visualising the world-wide web with generalised similarity analysis -
J. Kleinberg (1999)
Authoritative sources in a hyperlinked environment -
(1998)
The presentation of self in WWW home pages -
A. Broder, Ravi Kumar, F. Maghoul, P. Raghavan, S. Rajagopalan, Raymie Stata, A. Tomkins, J. Wiener (2000)
Graph structure in the WebComput. Networks, 33
-
Chaomei Chen, J. Newman, R. Newman, R. Rada (1998)
How did university departments interweave the Web: A study of connectivity and underlying factorsInteract. Comput., 10
-
(2001)
Accessed -
Robert Miller, K. Bharat (1998)
SPHINX: A Framework for Creating Personal, Site-Specific Web CrawlersComput. Networks, 30
-
M. Theimer, K. Lantz (1988)
Finding idle machines in a workstation-based distributed system[1988] Proceedings. The 8th International Conference on Distributed
-
P. Ingwersen (1998)
The calculation of web impact factorsJ. Documentation, 54
-
H. Snyder, H. Rosenbaum (1999)
Can search engines be used as tools for web-link analysis? A critical viewJ. Documentation, 55
-
J. Bar-Ilan (2000)
The Web as an information source on informetrics? A content analysisJ. Am. Soc. Inf. Sci., 51
-
(1999)
Mercator: a scalable -
M. Thelwall (2000)
Web impact factors and search engine coverageJ. Documentation, 56
-
Tham Chun (1999)
World wide web robots: an overviewOnline Inf. Rev., 23
-
F. Monrose, P. Wyckoff, A. Rubin (1999)
Distributed Execution with Remote Audit -
M. Ferris, M. Mesnier, J. Moré (2000)
NEOS and Condor: solving optimization problems over the InternetACM Trans. Math. Softw., 26
-
(1998)
Socio-economic impact of the internet in the academic research environment -
Stephanie Haas, Erika Grams (1999)
Structure : A Discussion of Four Questions Arising from a Content Analysis of Web Pages -
M. Thelwall (2001)
Extracting macroscopic information from Web linksJ. Assoc. Inf. Sci. Technol., 52
-
M. Thelwall (2001)
Results from a web impact factor crawlerJ. Documentation, 57
-
(2001)
Data collection on the web for informetric purposes – a review and analysis -
B. Kelly (2000)
Web Watch: A Survey of Links to UK University Web Sites -
Junghoo Cho, H. Garcia-Molina (2000)
The Evolution of the Web and Implications for an Incremental Crawler -
S. Brin, Lawrence Page (1998)
The Anatomy of a Large-Scale Hypertextual Web Search EngineComput. Networks, 30
-
David Gibson, J. Kleinberg, P. Raghavan (1998)
Inferring Web communities from link topology -
Neel Sundaresan, Jeonghee Yi (2000)
Mining the Web for relationsComput. Networks, 33
-
Alastair Smith (1999)
A Tale of Two Web Spaces: Comparing Sites Using Web Impact Factors.Journal of Documentation, 55
-
I. Middleton, M. McConnell, Grant Davidson (1999)
Presenting a model for the structure and content of a university World Wide Web siteJournal of Information Science, 25
-
J.P.H. Burden, M. Jackson (1999)
WWLib-TNG new direction in search engine technology -
Mark Overmeer (1999)
My personal search engineComput. Networks, 31
- Publisher
- SAGE
- Copyright
- Copyright © by SAGE Publications
- ISSN
- 0165-5515
- eISSN
- 1741-6485
- DOI
- 10.1177/016555150102700503
- Publisher site
- See Article on Publisher Site
Abstract
The content of the web has increasingly become a focus for academic research. Computer programs are needed in order to conduct any large-scale processing of web pages, requiring the use of a web crawler at some stage in order to fetch the pages to be analysed. The processing of the text of web pages in order to extract information can be expensive in terms of processor time. Consequently a distributed design is proposed in order to effectively use idle computing resources and to help information scientists avoid the need to employ dedicated equipment. A system developed using the model is examined and the advantages and limitations of the approach are discussed.
Journal
Journal of Information Science – SAGE
Published: Oct 1, 2001