e-Society Project

Computer Science Div., School of Science and Engineering, Waseda Univ.
Yoichi MURAOKA and Hayato YAMANA

If you have any problems or questions、please send the e-mail to desk@yama.info.waseda.ac.jp.
(Do not send any e-mail to me, directly)

About the Project

　The research project "Technologies for the Knowledge Discovery from the Internet" is one of the year 2003 leading projects of Ministry of Education, Culture, Sports, Science and Technology, Japan. The project contractor is Waseda university. The project goal is to gather all the Web pages in the world efficiently and to apply data mining to the gathered Web pages to discover the Knowledge. The following is the detailed sub goals.

1. R&D on new Web page crawler
　Gather all the Web pages (about 12 billion pages) and keep their freshness in one month in average.

2. R&D on discovering the knowledge
　Discover the knowledge that the user want to know.

In the project, 30 Web crawlers are running on 3 different sites. The detailed manner of the crawlers is described as bellow. We run the crawlers attentively not to disturb your Web sites. However, if you have any troubles, please let us know. We will get rid of any problems as soon as we can.

As for the outcomes of the project, we will distribute them widely as research papers. Moreover, the gathered Web pages are not distributed to others. The gathered Web pages are just used to analyze the links and the keywords to find out knowledge.

About our Web Crawler

• From the top page, 15 stratum Web pages are gathered at maximum.
• The interval of the GET command is 10 to 20 second.(as is 2005/2)
• Only one crawler access to your Web site. However, if your Web site has aliases, some crawlers may access to your Web site simultaneously.
• Never access to the same URLs in one term of gathering
• The crawler extracts the URL written in the following TAGs.
  • <A HREF=...> .... </A>
  • <AREA HREF=...>
  • <LINK HREF=...> <LINK SRC=...>
  • <FRAME SRC=...>
  • <IFRAME SRC=...>
• Binary data are not gathered such as zip, lzh, tar, tgz, gz, z, jpg, jpeg, gif, tiff, tif, mpg, mpeg, au, avi, wav, mov, mid, img, rpm, exe, com, bin, dvi, ra, log, pdf, xls, doc.
• The crawler is based on INTERNET DRAFT(robots.txt,Robot META Tag)(http://www.robotstxt.org/wc/robots.html）
• robots.txt will be cached by our robots in 24 hours.
• robots.txt is re-crawled every 6 hours.