IncidentSearchCrawlerErrors

Severity: warning · Fires after: 1h · Rule: central/prometheus/rules/visibility.yml

What it means

More than 10% of a search crawler's requests to the zone in the last 24h were answered 403, 429 or 5xx at the Cloudflare edge. Sustained, this is how a site gets de-indexed.

Expression

(
  sum by (zone, bot) (cloudflare_bot_requests_status{bot=~"googlebot|bingbot|yandexbot", status=~"403|429|5.."})
  / sum by (zone, bot) (cloudflare_bot_requests_status{bot=~"googlebot|bingbot|yandexbot"})
) > 0.10
and sum by (zone, bot) (cloudflare_bot_requests_status{bot=~"googlebot|bingbot|yandexbot"}) >= 20

• Source is Cloudflare's sampled adaptive dataset, hence the 20-request floor.
• Trailing-24h gauges: the alert lingers up to a day after the cause is fixed.
• Zone-level (wikantik.com, which includes the wiki) — the free plan's data here is not split per hostname in this metric.
• The bot is identified by User-Agent, so a scraper claiming to be Googlebot and being blocked for it also counts. Check Cloudflare's verified-bot view before changing a rule.

Reading the status

Status	Meaning	Look at
403	WAF / Bot Fight Mode / Access rule refused it	Cloudflare Security → Events
429	rate limiting	Cloudflare rate-limit rules; origin limiter
5xx (502/520–530)	origin or tunnel failure	cloudflared journal: {job="cloudflared"}; up{job="wikantik"}

First checks

curl -sG 'http://192.168.0.5:9090/api/v1/query' \
  --data-urlencode 'query=sum by (bot,status) (cloudflare_bot_requests_status{bot=~"googlebot|bingbot|yandexbot"})'

Or the What crawlers were served table on the wikantik-traffic dashboard.

How to clear

• 403 — find the rule in Cloudflare Security → Events that matched the crawler and exempt verified bots from it (Bot Fight Mode has no allowlist; a custom WAF rule with cf.client.bot does).
• 429 — raise or scope the rate-limit rule so verified crawlers are not counted, or fix the origin limiter that is returning it.
• 5xx — restore the origin or tunnel (IncidentServiceDown, IncidentEdgeVhostDown); nothing to do at the edge.

The gauge is a trailing 24h window, so the alert resolves on its own up to a day after the fix. Confirm recovery from the same table rather than silencing.

Why this exists

During the September 2026 re-crawl the zone served 1,525 429s in a day and there was no way to tell whether Googlebot was among the recipients: cloudflare_bot_requests carried only the bot family, and wiki traffic bypasses the edge nginx log.