Modify to allow comma-separated list of user agents #26
Conversation
Take a comma-delimited list of user agents and pass them as a vector.
My best understanding of what the code actually does with a vector of user agents is treat it as if all the rules applying to any of the user agents are collapsed into a single ruleset applying to all user agents
Update comments to include possibility of passing in a vector of user agents
|
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
|
@googlebot I fixed it. |
|
CLAs look good, thanks! ℹ️ Googlers: Go here for more info. |
Tidy up code to Google C++ standards
|
@garyillyes I remembered about this and wondered if you'd had a chance to take a look at it? I was reminded by the launch of Bing's new robots.txt checker... |
The underlying code accepts a vector of user agents (which, based on conversations with @garyillyes is how e.g.
googlebot-imageworks - running theAllowedByRobotsmethod against bothgooglebotandgooglebot-imageuser agents). Before this change, the wrapper inrobots_mainonly took a single user agent argument and passed it as a single element vector.Gary suggested that in order to enable the project to replicate the behaviour of googlebots like the images crawler, I submit a pull request to enable
robots_mainto accept a comma-separated list of user agents (likegooglebot,googlebot-image) that should then be passed toAllowedByRobotsas a vector.This pull request includes that change as well as changes to comments throughout the project for clarity / correctness in regards to this change.
I have also included some new tests on this new functionality one of which currently fails. It is based on the explicit worked example in the documentation.
It isn't clear to me whether I've got a bug in my change, the parser is wrong, or the documentation is wrong (this same description of how things should work appears in many places throughout the robots.txt help text) so I have currently submitted the pull request with a failing test hoping that we can clarify during the review process. I hope that's the right approach - I'm not very familiar with submitting to open source projects.