About Our Crawler
Yamana Laboratory.
Computer Science Div., School of Fundamental Science and Engineering, Waseda Univ.
Computer Science Div., School of Fundamental Science and Engineering, Waseda Univ.
About this project
Our research project is to analyze web page texts.
Out crawler's manner is described as below. Out crawler pays attention to reduce your web server's burden. However, if you have any troubles, please let us know. We will try to solve the problem as soon as possible.
About our crawler
- It extracts URLs described in a HTML and follows them.
- The interval of GET command for a domain in more than 5 seconds.
- The User-Agent name includes 'YamanaLab-Robot'.
- If you want to disallow our crawler to visit your site, please place '/robots.txt' as belows.
User-Agent: YamanaLab-Robot
Disallow: /
Inquery
If you have any problems or questions about our web crawler, please send an e-mail to
When you send us an e-mail, please include your web server's URL.
Thank you for your cooperation.