Documentation

Bots and crawlers

Crawlers and automated tools load your pages, and their browsers send reports like a visitor's would. This rule recognizes those reports by their user agent. It is off for now: bot reports appear in your lists and notify as usual.

State: Off

What it matches

Reports of every type whose user agent belongs to a bot, a crawler or an automated tool. That includes search engine crawlers such as Googlebot, Lighthouse runs and any user agent that names a bot.

The same judgement drives the Bots Only filter on the Reports list and the bot field of every report over MCP, so you can see today what the rule would filter.

Why it's noise

No visitor saw these errors. A crawler skips scripts, blocks resources and opens pages in an order no person would, so its reports rarely point at something your visitors meet.

What it can hide

  • Your own Lighthouse and CI runs. If you test with Lighthouse, its reports would be filtered, including a real CSP violation the run found.
  • A problem that only bots hit, such as a page that breaks for a crawler and so drops out of search.

See the filtered reports

While the rule is off, filter the Reports list to Bots Only, or pass bot: true to list_reports. Once it is on, pick Show filtered or pass include_filtered: true; each report's noise_rule names bot. See Noise Rules.