Web crawling

Notes:

Web crawler requirements

Robustness

Handle malicious and faulty pages

Politeness

Rate limit

Notes:

Audience question

Web crawler requirements

Distributed

Scalable, efficient

Quality

Crawl "good" pages more frequently

Freshness

Keep index up-to-date

Extensible

Data formats, protocols

Notes:

Audience question

Crawling

Notes:

URL Queue

aka. Crawl frontier
Priority queue

$$\text{priority} = f(\text{quality}, \text{importance}, \text{change rate})$$

Notes:

Distributed Crawling

Partition by domain
Cache DNS
Locality

Notes:

Link graph

Notes:

Link graph

Connectivity servers / indices
Store web graph, in- and out-links
Support graph queries: in- / out-links, in- / out-degree, traversal
Used for link analysis, etc.

Notes: