Add Chrome-Lighthouse to Google crawlers list #160
Conversation
|
Thank you! |
|
Hey @aydinkazim , I can't seem to find where this User Agent is documented. What I found is GoogleChrome/lighthouse#14917 and it seems that the User Agent has been removed. I need a documentation reference. |
|
Hi again 👋 You're right, the official documentation shows that the Chrome-Lighthouse user agent has been removed, as mentioned in GoogleChrome/lighthouse#14917. However, based on the current logs from my bot, it's still utilizing the Chrome-Lighthouse user agent. Here's a snippet from my logs: I, [2024-10-24T16:20:06.053244 #208883] INFO -- :
REAL BOT IP: 74.125.208.7 AGENT: Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Mobile Safari/537.36 Chrome-Lighthouse
I, [2024-10-24T16:20:06.539276 #208883] INFO -- :
REAL BOT IP: 74.125.208.8 AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Safari/537.36 Chrome-Lighthouse
I, [2024-10-24T16:20:06.704677 #208883] INFO -- :
REAL BOT IP: 74.125.208.7 AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Safari/537.36 Chrome-Lighthouse
I, [2024-10-24T16:20:06.878626 #208883] INFO -- :
REAL BOT IP: 74.125.208.7 AGENT: Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Mobile Safari/537.36 Chrome-Lighthouse
I, [2024-10-24T16:20:19.868516 #208883] INFO -- :
REAL BOT IP: 74.125.208.7 AGENT: Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Mobile Safari/537.36 Chrome-LighthouseIt seems that despite the removal in official references, the bot is still functioning with this user agent. I will continue monitoring this, but it may be a transitional state from Google Lighthouse. |
|
Thank you for confirming. It does not look from the issue that they would consider reintroducing it back, but let's see. I do not see any harm in adding it to Legitbot meanwhile. I am going to publish a release by the weekend. |
|
Thank you for the update! I completely understand, and I agree that reintroducing it doesn't seem likely based on the current discussions. Looking forward to the upcoming release! |
|
Published as |
|
I'm not sure if the Lighthouse user agent should be here, as the Lighthouse tool can be legitimately used by other companies that aren't Google, and this will cause the Rack Attack example on the README to block Lighthouse requests. Context: I upgraded legitbot as part of a change, and discovered that our Calibre (which uses Lighthouse) snapshots started to fail, as we use Rack Attack and legitbot to deny traffic to anything claiming to be a bot that they're not. @aydinkazim is there any particular reason why you want Lighthouse to be in Legitbot? |
|
@chendo I agree that it makes sense to remove it from Legitbot. |
If you check your site using https://pagespeed.web.dev/ and look at your bot logs, you will see "Chrome-Lighthouse." Based on this, I shared my opinion as I mentioned above. Therefore, our pages are also being blocked during the testing phase. |
No description provided.