dynamic contents

[vajrabisj]

how to use this to catch those dynamic contents? eg, some contents of li under ul cannot be downloaded unless you click the link, etc.

[svetlyak40wt]

If you mean sites which use client-side JavaScript to update content, then you have to use something like https://github.com/copyleft/cl-webdriver-client/ instead Scrapycl's fetch generic function. Probably a webdriver support might be added to the Scrapycl in future.

[vajrabisj]

Thanks. That looks a good library and I will try.

but for my case, when I downloaded the file there is no any li under the ul element. But when I browsed same page online and review through development tools I can see several li under the ul. Don’t know how to scrape such page, it’s a shtml file.

[svetlyak40wt]

That is why do you need to use a Webdriver. It downloads the page using a real browser with javascript enabled.

[vajrabisj]

Understood, thx!