SniffaprezziBot — Crawler documentation

This page is intentionally provided in English to be accessible to webmasters and CDN operators globally. Italian summary at the bottom.

Bot identity

• Name: SniffaprezziBot

• Operator: Sniffaprezzi.it (Italy)

• Website: https://www.sniffaprezzi.it

• Documentation (this page): https://www.sniffaprezzi.it/about-bot

User-Agent string

The bot identifies itself with the following exact User-Agent header:

SniffaprezziBot/1.0 (+https://www.sniffaprezzi.it/about-bot)

Webmasters can reliably block or rate-limit the bot by matching on the substring SniffaprezziBot at the start of the User-Agent.

Additional identification headers

Every outbound request also carries the following headers:

• From: bot@sniffaprezzi.it — abuse / contact mailbox

• X-Sniffaprezzi-Bot: 1.0 — protocol version

Authentication (Web Bot Auth, RFC 9421)

Outbound requests are cryptographically signed via RFC 9421 HTTP Message Signatures using an Ed25519 keypair, in compliance with the Cloudflare Web Bot Auth specification.

The public key is published as a JSON Web Key Set at:

https://www.sniffaprezzi.it/bot-auth-directory

The directory response is itself signed; the keyid in each request signature matches the JWK thumbprint of the published key.

Purpose and behaviour

SniffaprezziBot is a benign price-comparison crawler. It fetches publicly accessible product pages and images for the sole purpose of populating Sniffaprezzi.it, a consumer-facing price comparison service. The bot targets:

• Product images (image/* MIME types) for caching and display on Sniffaprezzi.it product detail pages.

• Public product metadata explicitly exposed via merchant feed URLs.

SniffaprezziBot does NOT:

• Crawl private or authenticated content

• Submit forms

• Bypass paywalls, CAPTCHAs, or anti-bot mechanisms

• Test or attempt to exploit vulnerabilities

• Scrape personal data, user reviews, or pricing for resale

• Spoof user-agents of other bots (Googlebot, Bingbot, etc.)

Rate limiting and politeness

• Per-host concurrency: 1 request at a time

• Per-host throttle: minimum 350 ms between consecutive requests

• Connection timeout: 10 s · Total timeout: 25 s

• Retry policy: up to 3 attempts with exponential backoff (1 s, 3 s, 9 s)

• Cooldown: if a host returns 3+ consecutive errors (timeout / 403 / 429 / 503), the bot pauses outbound traffic to that host for the remainder of the run and resumes at the next scheduled window (typically 10 minutes later)

• HTTP version: HTTP/2 where supported, falling back to HTTP/1.1

• Max payload: 5 MB per resource — larger downloads are aborted client-side

• Concurrency budget: overall, the bot generates well under 0.1% of typical merchant traffic

Compliance with robots.txt

SniffaprezziBot fully respects RFC 9309 (the Robots Exclusion Protocol). The bot reads robots.txt before crawling and honours both Disallow and Crawl-delay directives, as well as Retry-After response headers.

To block the bot specifically:

User-agent: SniffaprezziBot
Disallow: /

IP ranges

SniffaprezziBot crawls from a shared hosting infrastructure provided by our hosting partner. IP ranges are not pinned; for cryptographic verification, please rely on the Web Bot Auth signature on every request rather than IP allowlisting.

Opt-out and abuse

To report abuse, request opt-out outside of robots.txt, or coordinate technical issues:

• Abuse / opt-out: abuse@sniffaprezzi.it

• Technical contact: bot@sniffaprezzi.it

Opt-out requests are honoured within 24 hours of receipt. We maintain an internal block-list of merchants who have opted out and that list is loaded by the crawler before each run.

Data retention

Fetched images are cached as resized WebP copies (400×400 px) for the purpose of serving them on Sniffaprezzi.it. Original URLs and image hashes are stored for delta detection. We do not retain raw HTML snapshots of merchant pages.

Privacy

SniffaprezziBot does not collect personal data. Outbound requests carry only the headers listed above and do not transmit cookies, tokens, or end-user identifiers. See our privacy policy for further details.

Change log

• 1.0 — initial public release.

Riepilogo (Italiano)

SniffaprezziBot è il crawler di Sniffaprezzi.it, comparatore di prezzi consumer. Scarica esclusivamente immagini prodotto pubbliche per la cache locale del comparatore. Identificazione via User-Agent SniffaprezziBot/1.0, autenticazione crittografica tramite Web Bot Auth (RFC 9421, Ed25519). Throttle: 1 connessione concorrente per host, pausa di 350 ms tra richieste, cooldown automatico in caso di errori ripetuti. Rispetta robots.txt e Retry-After. Per bloccare il bot o segnalare abusi: abuse@sniffaprezzi.it.