Using Selenium for loading dynamic content #1092
-
|
Hello all, I've been trying to get SC to use the Selenium RemoteDriver HTTP protocol but I can't seem to make it work. The configuration I'm using right now is as follows:
Articles / documents I've referred to -
What I've managed to do so far:1) SC + Browserless on local topology (Followed the first blog post to the "T")
With some additions to the The config is as follows: NOTE: Both Initiate the crawl: Which gives me the following output: From the above output, it's clear that the topology is working as expected. Time to use the latest version. 2) SC + Browserless on local topology (Upgrade SC version)
With SC v2.9.0, the Selenium library is upgraded from The following error is encountered: Looking at the error logs and reading up on Selenium 4, it seems as though the
Am I correct in the above assumption? Or is there some step that I'm missing which can make SC work with Selenium 4? |
Beta Was this translation helpful? Give feedback.
Replies: 6 comments 16 replies
-
|
Thanks for the explanation @dhaneshsabane, the tutorial in the blog is indeed based on an old version. |
Beta Was this translation helpful? Give feedback.
-
|
If you provide a PR @dhaneshsabane we are very happy to review and test :-) |
Beta Was this translation helpful? Give feedback.
-
|
For adding tests, we could have the tests spinning a container just like we do e.g. for OpenSearch. |
Beta Was this translation helpful? Give feedback.
-
|
Have just created a branch seleniumTest; not working yet but fixes some of the dependencies |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the starting point @jnioche. I was walking on a similar path with the changes. To test this locally, can you share the steps I can take to build the SC core jar locally which can then be used in my selenium-tutorial project? I'm more of a Python dev, so I'm having to figure out the tooling around Java to make it work. |
Beta Was this translation helpful? Give feedback.
-
|
mvn clean install and remember to change the version of sc-core in your project, will be 2.10-SNAPSHOT I expect are obtained from crawler-default.yml. For some reason the values are not overridden by the user config. I will remove them from the default. What you will need to have in selenium.capabilities is: There is also a line to remove from RemoteDriverProtocol.java So quite a few changes needed in the code, which will be done as part of the PR adding the test. I might release a new patch version soon. |
Beta Was this translation helpful? Give feedback.
-
|
I have updated netty in Storm. Next Storm will contain 4.1.96.Final or higher. It might be possible to shade selenium/netty with namespace relocation in the client jar as a temporary workaround, so we do not run into class collision with stuff provided by the Storm env. |
Beta Was this translation helpful? Give feedback.
-
|
Nice work @dhaneshsabane, thanks! I don't think the Java 11 workaround would work as it is for Selenium Grid but would still be worth exploring further. FYI I have pushed some changes to the PR which adds the tests. It fixes a couple of things but in particular you can test the protocol implementation without running a whole crawl:
Loads of logs but quite useful for testing the conf |
Beta Was this translation helpful? Give feedback.
-
|
@rzo1 - Thanks for the library upgrade in Apache Storm! That should really ease things out. @jnioche - Thanks for the updated PR! I've managed to implement the Java 11 workaround and it is able to connect to my remote web driver but the connection is killed instantly. I think this might be because the current SeleniumProtocol implements "pooling" of the driver connections using Here are the logs from the web driver: As you can see, the connection is closed right the next second. Corresponding logs from SC are below: I think switching to the Java 11 method makes sense as Selenium seems to be going that way. Looking at the official Docker containers from Selenium, they seem to be using the Java 11 HTTP client - How about keeping the current |
Beta Was this translation helpful? Give feedback.
-
|
I created a simple crawler with the archetype. Build Julien's branch locally and upgraded to latest Selenium. To workaround the netty issue, I upgraded the shade plugin to 3.5.0 (in the archetype) and added relocations: <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.5.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.apache.storm.flux.Flux</mainClass>
<manifestEntries>
<Change></Change>
<Build-Date></Build-Date>
</manifestEntries>
</transformer>
</transformers>
<!-- The filters below are necessary if you want to include the Tika
module -->
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
<filter>
<!-- https://issues.apache.org/jira/browse/STORM-2428 -->
<artifact>org.apache.storm:flux-core</artifact>
<excludes>
<exclude>org/apache/commons/**</exclude>
<exclude>org/apache/http/**</exclude>
<exclude>org/yaml/**</exclude>
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>io.netty</pattern>
<shadedPattern>org.example.shaded.io.netty</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>This is the selenium-related part of the crawler config selenium.addresses: "http://localhost:3000/webdriver"
selenium.implicitlyWait: 1000
selenium.pageLoadTimeout: 1000
selenium.scriptTimeout: 10000
selenium.capabilities:
# a brower name is required
browserName: "chrome"
# illustrates the use of the variable for user agent
# phantomjs.page.settings.userAgent: "$userAgent"
# ChromeDriver config
goog:chromeOptions:
args:
- "--disable-dev-shm-usage"
- "--no-sandbox"
- "--headless"
- "--disable-gpu"
- "--mute-audio"
# DelegatorRemoteDriverProtocol
selenium.delegated.protocol: "com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol"
Next, I started the chrome instance: docker run -p 3000:3000 browserless/chromeNext, I started without an actual crawl as described by Julien: This results in https://storm.apache.org/
robots allowed: true
robots requests: 1
sitemaps identified: 0
status code: 200
content length: 12533
fetched in : 746 msecGuess, this is working? The thing is, that the current impl isn't really error save. If a page takes more time to load (for example: https://www.hs-heilbronn.de/de), it will just fail with Session ID: 0b248b43072f94d64a515d3756f4d9e5
org.openqa.selenium.TimeoutException: timeout: Timed out receiving message from renderer: 0.004
(Session info: headless chrome=116.0.5845.96)
Build info: version: '4.12.1', revision: '8e34639b11'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-83-generic', java.version: '17.0.8.1'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Command: [0b248b43072f94d64a515d3756f4d9e5, get {url=https://www.hs-heilbronn.de/de}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 116.0.5845.96, chrome: {chromedriverVersion: 116.0.5845.96 (1a3918166880..., userDataDir: /tmp/browserless-data-dir-J...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:44259}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
Session ID: 0b248b43072f94d64a515d3756f4d9e5
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:200)
at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:133)
at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:52)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:191)
at org.openqa.selenium.remote.TracedCommandExecutor.execute(TracedCommandExecutor.java:51)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:602)
at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:304)
at com.digitalpebble.stormcrawler.protocol.selenium.SeleniumProtocol.getProtocolOutput(SeleniumProtocol.java:47)
at com.digitalpebble.stormcrawler.protocol.AbstractHttpProtocol$1Fetchable.run(AbstractHttpProtocol.java:254)
at java.base/java.lang.Thread.run(Thread.java:833)
The exception isn't nice, but says everything. I am wondering about the exception fron the reflection-based instance creation, but "okay". It's another story, but wonder if implementing some sort of back-off with an increasing timeout value until we reach a pre-defined maximum (or similar) would be a good thing. Increasing the value results in success btw: https://www.hs-heilbronn.de/de
robots allowed: true
robots requests: 1
sitemaps identified: 1
status code: 200
content length: 1718
fetched in : 1166 msectl;dr: Seems that upgrading Selenium to |
Beta Was this translation helpful? Give feedback.
-
|
Nice one! I will upgrade the version of Selenium in the PR which also adds the tests. Compared to the base version, it merely removes incompatible confs, those could be added explicitly to match older versions if needed. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good to me! 👍🏽 |
Beta Was this translation helpful? Give feedback.
-
|
Meanwhile, I guess the guava dependency used in storm-hdfs needs an upgrade too - |
Beta Was this translation helpful? Give feedback.
-
|
Won't be that easy as it means to upgrade hadoop. Guess you need to use the workaround for Guava too |
Beta Was this translation helpful? Give feedback.
-
|
No worries! For now, your workaround works well, even with a normal crawl. I'll deploy the fix to my server. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for reporting the issue and the detective work @dhaneshsabane, this is a great contribution |
Beta Was this translation helpful? Give feedback.
I created a simple crawler with the archetype. Build Julien's branch locally and upgraded to latest Selenium.
Next, started the headless chrome docker container and configured my crawler-conf.yaml
To workaround the netty issue, I upgraded the shade plugin to 3.5.0 (in the archetype) and added relocations: