Increase crawling speed of the crawler.

[msghasan]

HI Julien,

We are trying to speed up our crawling speed. Currently our crawling lack and the speed is around 1.5 urls per second and I want to increase it to 9 urls persec.

I have deployed storm crawler over kubernetes and using a node pool of 4 nodes of 3.92 cores each and memory of 29.72 gb each.

I am running 8 replicas supervisors with 1 worker in each supervisor

We are using selenium grid for crawling and running 100 chrome replicas over kubernetes.

We are using storm-crawler 2.7 and apache storm version 2.4

I need help to achieve the above mentioned speed.

Can you guide me into the direction how can I achieve this I will upload all the config documents required for you to analyse.

crawler-conf.yaml

config:
  topology.workers: 8
  topology.message.timeout.secs: 300
  topology.max.spout.pending: 500
  topology.debug: false

  fetcher.threads.number: 20
  fetcher.threads.per.queue: 1
  fetcher.server.delay: 1
  fetcher.server.delay.force: true
  fetcher.metrics.time.bucket.secs: 10
 

  kf.bootstrap.servers: localhost:9092
  kf.auto.offset.reset: earliest
  kf.consumer.seedurl.topic: SeedUrlsTopic
  kf.producer.urlstatus.topic: UrlStatus
  kf.producer.htmlsource.topic: HtmlSource


  topology.worker.childopts: "-Djava.net.preferIPv4Stack=true"


  topology.kryo.register:
    - com.digitalpebble.stormcrawler.Metadata
    - java.util.LinkedList
    - com.digitalpebble.stormcrawler.persistence.Status
    - java.util.concurrent.atomic.AtomicLong
    - org.apache.storm.generated.NodeInfo

  metadata.transfer:
    - projectId
    - seedUrlId

  metadata.persist:
    - _redirTo
    - error.cause
    - error.source
    - isSitemap
    - isFeed
    - fetch.statusCode
    - fetch.byteLength
    - fetch.loadingTime
    - fetch.timeInQueues
    - fetch.exception


  http.robots.file.skip: true


  http.robots.headers.skip: true


  http.robots.meta.skip: true

  http.agent.name: "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
  http.agent.version: "2.0"
  http.agent.description: "Chrome Browser"
  http.agent.url: "https://test.com"
  http.agent.email: "[email protected]"

  http.content.limit: 262144
  http.protocol.implementation: "codes.util.SeleniumGridImpl"
  https.protocol.implementation: "codes.util.SeleniumGridImpl"
  partition.url.mode: "byDomain"

  selenium.grid.address: http://localhost:4444/wd/hub

  selenium.capabilities.browserName: chrome


  redirections.allowed: true
  parser.emitOutlinks: false

  parsefilters.config.file: "parsefilters.json"
  urlfilters.config.file: "urlfilters.json"

  fetchInterval.default: -1

  fetchInterval.fetch.error: 600


  fetchInterval.error: 600


  max.fetch.errors: 3


  status.updater.use.cache: false


  textextractor.include.pattern:
    - DIV[id="maincontent"]
    - DIV[itemprop="articleBody"]
    - ARTICLE

  textextractor.exclude.tags:
    - STYLE
    - SCRIPT


  indexer.url.fieldname: "url"
  indexer.text.fieldname: "content"
  indexer.canonical.name: "canonical"
  indexer.md.mapping:
    - parse.title=title
    - parse.keywords=keywords
    - parse.description=description
    - domain=domain

  
  topology.metrics.consumer.register:
    - class: "org.apache.storm.metric.LoggingMetricsConsumer"
      parallelism.hint: 1
      argument: "yyyy-MM-dd"

  fetcher.max.crawl.delay.force: true
  http.trust.everything: true

storm.yaml

storm.zookeeper.servers:
     - "zookeeper-crawler-entrypoint.$NAMESPACE.svc.cluster.local"
   nimbus.seeds: ["nimbus-crawler-entrypoint.$NAMESPACE.svc.cluster.local"]

   supervisor.slots.ports:
     - 6700
   
   logviewer.port: 80
   storm.zookeeper.port: 2181

   worker.heap.memory.mb: 12083
   supervisor.memory.capacity.mb: 120832
   topology.worker.max.heap.size.mb: 12083

   topology.component.resources.onheap.memory.mb: 572
   worker.childopts: "-Xmx%HEAP-MEM%m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump"

es-conf.yaml

config:
  
  es.indexer.addresses: "localhost"
  es.indexer.index.name: "content"

  es.indexer.create: false
  es.indexer.bulkActions: 100
  es.indexer.flushInterval: "2s"
  es.indexer.concurrentRequests: 1


  es.metrics.addresses: "http://localhost:9200"
  es.metrics.index.name: "metrics"


  es.html.address: "localhost"


  es.status.address: "localhost"
  es.status.addresses: "http://localhost:9200"
  es.status.index.name: "status"



  es.status.routing: true
  es.status.routing.fieldname: "key"
  es.status.bulkActions: 500
  es.status.flushInterval: "5s"
  es.status.concurrentRequests: 1
  spout.ttl.purgatory: 60


  spout.min.delay.queries: 2000

  spout.reset.fetchdate.after: 120

  es.status.max.buckets: 10
  es.status.max.urls.per.bucket: 4

  es.status.bucket.field: "metadata.projectId"

  es.status.bucket.sort.field:
    - "nextFetchDate"
    - "url"

  es.status.global.sort.field: "nextFetchDate"


  es.status.max.start.offset: 500


  es.status.sample: false


  es.status.query.timeout: -1

  es.status.recentDate.increase: -1
  es.status.recentDate.min.gap: -1
  topology.metrics.consumer.register:
    - class: "com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer"
      parallelism.hint: 1

topology.yaml

name: "crawler"

includes:
  - resource: true
    file: "/crawler-default.yaml"
    override: false

  - resource: false
    file: "crawler-conf.yaml"
    override: true

  - resource: false
    file: "es-conf.yaml"
    override: true

  - resource: false
    file: "mongo-conf.yaml"
    override: true

  - resource: false
    file: "minio-conf.yaml"
    override: true

spouts:
  - id: "spout"
    className: "com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout"
    parallelism: 10
  - id: "kafka-spout"
    className: "codes.kafka.client.AddSeedUrlsSpout"
    parallelism: 1

bolts:
  - id: "filter"
    className: "com.digitalpebble.stormcrawler.bolt.URLFilterBolt"
    parallelism: 1
  - id: "partitioner"
    className: "com.digitalpebble.stormcrawler.bolt.URLPartitionerBolt"
    parallelism: 1
  - id: "fetcher"
    className: "codes.bolt.CustomFetcherBolt"
    parallelism: 80
  - id: "sitemap"
    className: "digitalpebble.stormcrawler.bolt.SiteMapParserBolt"
    parallelism: 1
  - id: "parse"
    className: "codes.bolt.CustomJSoupParserBolt"
    parallelism: 3
  - id: "status"
    className: "codes.bolt.CustomStatusUpdaterBolt"
    parallelism: 8
  - id: "status_metrics"
    className: "com.digitalpebble.stormcrawler.elasticsearch.metrics.StatusMetricsBolt"
    parallelism: 1
  - id: "extract_html"
    className: "codes.extract.html.ExtractHtmlBolt"
    parallelism: 4
  - id: "smart_filter"
    className: "codes.filter.SmartFilterBolt"
    parallelism: 1
  - id: "es_index"
    className: "codes.elasticsearch.persistence.ESIndexerBolt"
    parallelism: 1

streams:
  - from: "spout"
    to: "partitioner"
    grouping:
      type: SHUFFLE

  - from: "spout"
    to: "status_metrics"
    grouping:
      type: SHUFFLE

  - from: "kafka-spout"
    to: "status"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "status"

  - from: "partitioner"
    to: "fetcher"
    grouping:
      type: FIELDS
      args: ["key"]

  - from: "fetcher"
    to: "sitemap"
    grouping:
      type: LOCAL_OR_SHUFFLE

  - from: "fetcher"
    to: "status"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "status"

  - from: "sitemap"
    to: "parse"
    grouping:
      type: LOCAL_OR_SHUFFLE

  - from: "parse"
    to: "status"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "status"

  - from: "parse"
    to: "extract_html"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "extract_html"

  - from: "parse"
    to: "smart_filter"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "smart_filter"

  - from: "smart_filter"
    to: "es_index"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "index_filter"

  - from: "smart_filter"
    to: "status"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "status"

  - from: "extract_html"
    to: "es_index"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "index_html"

[rzo1]

My tests with Selenium grid revealed, that SC cannot handle that correctly atm. It blocks with the single instance URL used in the related bolt. There is an open issue for that.

It might be faster (atm) to specify multiple standalone selenium instances.

[msghasan]

@jnioche @rzo1 Its been 1 week, I am waiting for you guys to reply. Please help me its urgent. I might even change my approach if you guys think that config wise every thing is ok.

One more thing I would like to add is we are also taking screenshots of the webpage while we are fetching the source code using selenium driver.

[rzo1]

Gentle reminder: storm-crawler is an open source project, maintained entirely by volunteers, all of whom have day jobs that they normally work at.
That being said delays in answering questions, debugging things, etc. is totally normal because it is the maintaners free decision how they are going to spend their spare time. There is no SLA, etc. - if you are looking into commercial support with agreed SLA because it is "urgent" for your (business) use-case, I would recommend, that you look into related consultancy services.

That being said, here are some additional thoughts:

Dynamic web crawling is typically a bit slow due to the JS execution, etc. involved. It fully depends on the protocol impl.

I suspect, that the bottleneck in your topology is exactly this process:

• Is it also slow if running locally?
• How is the resource consumption of selenium grid? Are the queues full? Are the queues empty? Is every browser instance used?
• Have you tried to profile the entire application to see where the majority of time is consumed?

We don't have the full picture, so we cannot say anything about your specific impl.

[msghasan]

Thanks for the reply and definitely there are no bonding or SLA this is a community, that's why I requested with a please.
I am a volunteer myself to this project . It was more than 4 days after my last message that's why I messaged again. I will keep in mind not to bother you gusy very frequently.

Answering your Questions:

• Is it running also slow locally? (Yes it is running slow, we are using docker for running all our services so decided to test and run on google cloud)
• How is the resource consumption of selenium grid? Are the queues full? Are the queues empty? Is every browser instance used?(Yes it always runs in full blast)
  
• Have you tried to profile the entire application to see where the majority of time is consumed? I have not tried profiling the full application but tried profiling the screen capture part of selenium driver. which comes around generally from 6 to 22 seconds if its a long webpage. Even if I take average time to take screenshot as 12 seconds we are running 100 replicas which comes below 1 seconds.

This weekend I will push the changes for selenium-grid onto storm-crawler project.

If you see the below issue I am the same person commented there for building the selenium-grid code. Excuse me for expecting a quick reply.

#1023

[rzo1]

I would suggest profiling the application to check there the bottleneck is. Most likely, I would suspect that it is related to the selenium / selenium grid instance.

I don't know how you did implement the grid support (so seeing the actual protocl code would help), but given the queue size of 1 on the screenshot, I would suspect, that your fetcher threads are just waiting for the grid to return related results until the next webpage can be queued for rendering.

[msghasan]

I designed the protocol code in such a way that it will get the drivers only from a queue rather than queing the requests in selenium grid as it was crashing the grid after so many requests from crawler.

I will try to take some time out and push the code today.