Skip to content

Commit

Permalink
List supported crawlers [skip ci]
Browse files Browse the repository at this point in the history
  • Loading branch information
alaz committed Dec 21, 2016
1 parent 473b661 commit 4175327
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,15 @@ Rack::Attack.blocklist("fake Googlebot") do |req|
end
```

## Supported

* [Google crawlers](https://support.google.com/webmasters/answer/1061943)
* [Yandex robots](https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.xml)
* [Bingbot](https://blogs.bing.com/webmaster/2012/08/31/how-to-verify-that-bingbot-is-bingbot/)
* [Baidu spider](http://help.baidu.com/question?prod_en=master&class=498&id=1000973)
* [DuckDuckGo bot](https://duckduckgo.com/duckduckbot)
* [Facebook crawler](https://developers.facebook.com/docs/sharing/webmasters/crawler)

## Issues, problems, plans

* Rails middleware
Expand Down

0 comments on commit 4175327

Please sign in to comment.