New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Obeying robots.txt? #9

Open

giorgio79 opened this issue May 24, 2014 · 3 comments

giorgio79 commented May 24, 2014

Any plans for robots.txt compliance?

Owner

dakrone commented May 24, 2014

Itsy already looks at robots.txt, see: https://github.com/dakrone/itsy/blob/master/src/itsy/core.clj#L133

Author

giorgio79 commented May 24, 2014

Nice. Do you plan on implementing crawl-delay too? http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive
Or just a general delay setting?

Owner

dakrone commented May 30, 2014

I think supporting the crawl-delay parameter would be the best, I'll add it to the todo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment