Crawler simply stops

[mrtnmueller]

I have an issue where the crawler stops without giving any hint about why. Seems to be non-deterministic, always at another point during the crawl process.

I have some middlewares (retry, proxy, user agent header) and a redis queue but I don't think they create the problem.

The real problem seems to be that I add URLs dynamically in the observer using $crawler->addToCrawlQueue in the crawled function.

Do I need to call something else to make the crawler crawl all URLs on queue?

[mrtnmueller]

It seems that whenever a curl-multi child process "dies" (for whatever reason, they die without giving any hint) it will never be restarted and when all of them are dead, the whole crawler just dies. The observer function finishedCrawling is not called in that moment, because there are still urls in the queue.