v1.19: adds DuckAssistBot, improves automation

What's Changed

• Update robots.json with Claude respect link by @lxjv in #43
• Sort the content of robots.json by keys, case-insensitively by @isagalaev in #46
• Update the FAQ by @glyn in #45
• log the diff in the update actions by @fabianegli in #47
• PHP diagnostics by @glyn in #49
• Fix typo and trigger rerun of main job by @glyn in #50
• PHP to Python plus tests and stuff by @fabianegli in #51

New Contributors

• @lxjv made their first contribution in #43
• @isagalaev made their first contribution in #46
• @fabianegli made their first contribution in #47

Full Changelog: v1.18...v1.19

v1.18: adds ISSCyberRiskCrawler

Adds ISSCyberRiskCrawler. Thanks to Brian at Skewray Research for calling this one out!

v1.17: add Sidetrade indexer bot + other updates

What's Changed

• Add Sidetrade indexer bot (Thank you to Brian at Skewray Research for calling this one out.)
• Add instructions for AI bot blocking on Vercel by @cramforce in #36
• Update README.md by @cityrolr in #41
• feat: make CCBot entry more accurate by @wumpus in #42
• Add an additional resource - README.md by @urvish-p80 in #38

New Contributors

• @cramforce made their first contribution in #36
• @cityrolr made their first contribution in #41
• @wumpus made their first contribution in #42
• @urvish-p80 made their first contribution in #38

Full Changelog: v1.16...v1.17

v1.16: adds iaskspider/2.0

Thank you to Brian at Skewray Research for calling this one out.

What's Changed

• Improve main workflow by @nisbet-hubbard in #35

Full Changelog: v1.15...v1.16

v1.15: adds AI2Bot

Thank you to Brian at Skewray Research for calling this one out.

Full Changelog: v1.14...v1.15

v1.14: adds Webzio-Extended

What's Changed

• Adds the Webzio-Extended bot
• only build on changes to robots.json by @jbowdre in #29
• Add FAQ by @glyn in #31
• Tracking Dark Visitors Automatically by @ChenghaoMou in #32
• Add links for reporting and FAQ to README.md by @nisbet-hubbard in #33

New Contributors

• @jbowdre made their first contribution in #29
• @ChenghaoMou made their first contribution in #32

Full Changelog: v1.13...v1.14

v1.13: corrects agent name

What's Changed

• Fix Imagesift user agent by @jsheard in #27
• Add Cloudflares first-party scraper blocking to FAQ by @jsheard in #28

New Contributors

• @jsheard made their first contribution in #27

Full Changelog: v1.12...v1.13

v1.12: adds ICC-Crawler

What's Changed

• Add last line of defence to FAQ by @nisbet-hubbard in #24
• Add ICC-Crawler by @mirium999 in #25

New Contributors

• @mirium999 made their first contribution in #25

Full Changelog: v1.11...v1.12

v1.11: adds PetalBot and facebookexternalhit

What's Changed

• Add PetalBot (and facebookexternalhit?) by @nisbet-hubbard in #22

Full Changelog: v1.10...v1.11

v1.10: adds Scrapy

Thank you to Brian at Skewray Research for calling this one out.

Full Changelog: v1.9...v1.10