Max crawl depth not working #482
Unanswered
sasa-andjelic-nqode
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
When I start a crawl without setting the max crawl depth, the crawler scans for links on all pages.
But when I set the max crawl depth to some int value (for example 100), the crawler only scans the first layer of links and crawls those.
I took a peek into Crawler->addToDepthTree() implementation and I see that if the max depth is null any url is instanciated as node. The rest of the conditions inside the method implementation do not cover a case for child links.
I also didn't notice any increment for current depth of the link.
It looks like max depth was never implemented completely. Is it that or am I perhaps missing something obvious which I should pay attention to?
Beta Was this translation helpful? Give feedback.
All reactions