The Bots & Agents Page

What the page is for

"Bot traffic" is not one thing. Googlebot indexing your catalogue, a scraper pretending to be Googlebot, an AI shopping agent trying to check out, and a declared monitoring bot all need different treatment. The Bots & agents page (under Monitor) splits them apart so you can decide policy per group instead of guessing.

The four groups

• Verified crawlers — traffic that claims to be a known crawler (Googlebot, Bingbot, etc.) and passes verification. Usually welcome; blocking it hurts your SEO.
• Impersonation — traffic that claims to be a known crawler but fails verification. There is no innocent explanation for a fake Googlebot: real crawlers always verify. This group feeds the Verified Bot Impersonation detector, which opens an alert when fake-crawler traffic crosses its floor.
• AI agent requests — traffic from declared AI agents (assistants and shopping agents browsing on a person's behalf). The page lists which agents have been seen. The AI Agent Traffic detector alerts specifically when agents attempt transactions (POST requests: add to cart, checkout, login), because reading a product page and attempting a purchase deserve different policies.
• Other bots — declared bots that aren't verified crawlers or known AI agents: monitors, SEO tools, unclassified automation.

Each group drills into the networks behind it, so "impersonation traffic is up" turns into "and it comes from these two hosting providers" in one click.

Sensitive paths touched

Every agent, crawler, and impersonator row carries a sensitive-paths flag: a red marker with checkout / auth / cart chips when that bot's traffic hit an SFCC checkout, account, or cart controller. It is computed independently of volume, so a bot that is millions of image requests but pokes a checkout controller thirty times still lights up. This is the signal a raw total hides: "139k fake Googlebot" reads as image scraping until the flag shows a handful of those requests landed on your payment pages.

Treat the flag as a prompt to look, not a verdict. Real search crawlers do render some pages, so a verified crawler touching a cart endpoint can be legitimate. The point is that you see it and decide.

Drilling into what a bot scanned

Click any row to expand it. Edge shows:

• Top paths the bot hit, each with its request count and, where relevant, its POST volume and error (4xx/5xx) count. POST volume on a crawler or agent means it tried to transact, not just read; errors mean it is probing endpoints that reject it.
• Sensitive paths touched — the exact checkout / auth / cart controllers behind the row flag, listed even when their volume is far too low to appear in the top paths.

For an impersonator row, the drawer scopes to the fake (unverified) half only, so you see what the spoofing traffic specifically went after, separated from any genuine crawler on the same name.

The drawer shows the busiest paths; the long tail collapses into *other. When you need every URL a suspicious client touched, open its network's detail page and run the forensic IP sampler.

What to do with each group

• Verified crawlers: leave alone unless crawl volume is causing origin load.
• Impersonation: treat as hostile. The alert carries a drafted rule; challenge-first is safe because a real crawler never solves a challenge from a fake source anyway.
• AI agents: a policy decision, not automatically hostile. Some merchants welcome agent traffic (it converts), others challenge transactions from agents. Edge tells you the volume and lets you decide.
• Other bots: audit occasionally; ignore known-good monitors via the usual pin/ignore controls on Offenders.

Requirements

Crawler verification depends on the BotScoreSrc field in your Logpush job. If it's missing, the page shows a banner and impersonation detection is inactive until you add the field. See HTTP Methods and Required Logpush Fields.