international craigslist URL

[jonpolak]

for cities outside the US craigslist redirects away from the .org domain. e.g. vancouver.craigslist.ca - any way to configure this?

[WIRUT]

@jonpolak I was able to get it to work outside of the US. Without knowing what you have as your arguments, can't really see where you're going wrong.

[jonpolak]

what were the settings you used in your test? @WIRUT

[jonpolak]

@WIRUT are you using digitalocean as your cloud provider? I find craigslist super slow when I use my own vpn server hosted at digitalocean. Maybe they are limiting digitalocean connections since lots of scrapers host their bots there?

[jonpolak]

@VikParuchuri CONFIRMED craigslist blocks EC2 and DigitalOcean IPs

[WIRUT]

@jonpolak I was able to test it with Calgary as well as using Amazon's EC2 services. I didn't bother using digital ocean. I'm not sure which settings you want. I did set SITE = "Calgary", and I think I bypassed the area and just hardcode the AREA in for this test instance.

Amazon EC2 was nothing different either.