{web reveal}

Bot Ethics & Transparency Policy

Last updated: May 1, 2026

Webreveal operates as a "Good Bot" — a transparent, ethical infrastructure-auditing tool designed to help developers, agencies, and security researchers understand the modern web's technology landscape. Like search-engine crawlers, we follow strict ethical crawling guidelines and publicly document them here.

1. What the Web Reveal Scanner Does

When you submit a URL, Web Reveal makes a single HTTP request to that public URL — the same request any browser or search-engine crawler would make. We inspect only what the web server voluntarily returns in its public response:

  • HTML page source (client-side markup and inline scripts)
  • HTTP response headers (e.g. Server, X-Powered-By, Content-Security-Policy)
  • DNS records associated with the domain
  • Publicly linked asset filenames (e.g. JavaScript bundle paths used to detect framework signatures)

We derive technology signals exclusively from these publicly available, client-side signals. We do not access server-side configuration, databases, source-code repositories, admin panels, or any content that requires authentication.

2. Respectful Scanning

Web Reveal is designed to have the smallest possible footprint on the websites it scans:

  • Single-request scans: each scan generates one HTTP request per URL submitted — we do not crawl links or spider entire sites.
  • Low frequency: we do not perform automated or high-frequency polling of any domain.
  • No bandwidth impact: our request profile is indistinguishable from a standard browser visit and does not materially impact host bandwidth or server performance.
  • Standard User-Agent: our scanner identifies itself clearly so server operators can recognise and, if they wish, block our requests.

3. Robots.txt Compliance

Web Reveal respects robots.txt directives. If a website's robots.txt file disallows scanning of the requested path, the scan will be blocked and no data will be collected for that URL. Website operators who wish to opt out of Web Reveal scanning entirely may add a Disallow rule for our user-agent to their robots.txt file. Contact jared@webreveal.io if you need assistance with this.

4. Public Data Only

We only analyse publicly accessible, client-side information — data that any visitor to the site can see. We do not:

  • Attempt to access password-protected pages, login portals, or restricted areas
  • Probe for configuration files such as .env, wp-config.php, or similar
  • Attempt to access server-side code, databases, or back-end infrastructure
  • Collect or store personally identifiable information (PII) about website visitors or end users
  • Intercept user sessions, form submissions, chat interactions, or any dynamic user data

5. Integrity & Security

Web Reveal will never attempt to:

  • Circumvent security measures, firewalls, or rate-limiting systems
  • Solve or bypass CAPTCHAs or anti-bot challenges
  • Exploit vulnerabilities, inject scripts, or interact with site forms or APIs
  • Spoof or misrepresent its identity in HTTP requests
  • Decompile, reverse-engineer, or otherwise access any proprietary back-end logic

Our system is designed purely for passive, read-only observation of public HTTP responses — nothing more.

6. Purpose & Intended Use

Web Reveal provides real-time technology forensics for:

  • Developers researching the tools and frameworks used across the web
  • Agencies auditing client or competitor technology stacks
  • Security researchers understanding infrastructure patterns and exposure
  • Businesses conducting legitimate competitive analysis of publicly observable technical choices

Web Reveal is not intended, and must not be used, as a tool for unauthorised access, targeted attack reconnaissance, or any purpose that violates applicable law. See our Terms of Service for the full acceptable-use policy.

7. Data Retention & Scan Reports

Scan results may be stored and displayed publicly as a factual record of a site's observable public-facing configuration at the time of the scan. Technology signals displayed in scan reports are derived solely from information the target web server included in its public response. If you are a website owner and wish to request a review or removal of a scan report, please see our Request Site Removal page.

8. Accountability & Contact

We take our responsibilities as a good-faith automated agent seriously. If you believe our scanner has acted contrary to this policy, or if you have questions about our scanning practices, please contact us:

We aim to acknowledge all enquiries within 2 business days.