Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WarcMiddleware is adding a 'Host' header to requests, creating redicrt loops. #3

Open
esegal opened this issue Apr 29, 2015 · 0 comments

Comments

@esegal
Copy link

esegal commented Apr 29, 2015

With warcmiddleware.py, doing 'scrapy fetch http://www.pcaobus.org' I get a redirect loop.
I've isolated the problem to the line "headers.setdefault('Host', netloc)" in _warcrec_from_scrapy_request

When removing it everything seems to be working fine. Also - I think the middleware doesn't intend to change the headers, but it does.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant