Custom Robots.txt
& Crawler Builder

Configure crawler directives, define search engine access permissions, specify XML sitemap paths, and manage crawl rates effortlessly.

Presets:

Default Access (User-agent: *)

Crawl-Delay (Seconds)

XML Sitemap URL

Disallowed Directories (One per line)

Block Specific AI Scrapers:

GPTBot (OpenAI) CCBot (Common) ClaudeBot Bytespider

robots.txt Output

RFC 9309 Ready

Real-time Robots Exclusion Protocol file builder

Formatted Directive Text

utf-8 plain text

Upload to root directory: /robots.txt Create XML Sitemap →

Directive Telemetry

Protocol Standard: RFC 9309 / REP

Root File Path: /robots.txt

Search Crawler Status: Standard Indexing

Directives Reference

Core Robots.txt Syntax & Rules

The Robots Exclusion Protocol (REP) is an RFC 9309 standard governing search engine crawl budgets and path indexing.

User-agent Directive

Target Crawler Family

Defines which crawler the following rules apply to. 'User-agent: *' targets all bots, while specific tokens target individual crawlers.

Disallow & Allow

Path Access Rules

'Disallow: /admin/' stops compliant crawlers from scraping folders. 'Allow: /' grants explicit access to sub-resources.

Sitemap Declaration

Index Acceleration

Declares the absolute URL of your XML sitemap, allowing search engines to discover all indexable pages in a structured manifest.

Crawl-Delay Rule

Crawl Rate Limiting

Supported by Bing and Yandex to prevent server spikes by enforcing a pause (in seconds) between consecutive page requests.

Active Crawler Defense

Ad review bots & scrapers ignore robots.txt.
Enforce bulletproof edge filtering with Cloaker.net.

While search engines respect robots.txt, ad reviewers (Meta, Google Ads, TikTok) and scrapers ignore text directives. Cloaker.net inspects hardware fingerprints and TLS ciphers at the edge in under 15ms.

Active Bot & Reviewer Interception

Sub-15ms Single-Hop Edge Redirection

Pre-Filtered Meta & Google Signatures

Dynamic Country, Device & ASN Routing

Start Free Trial Instant Setup · Zero Code Setup

Support & Guidance

Frequently asked questions

Common technical questions about crawler directives, disallow rules, and cloaking defense.

What is a robots.txt file?

A robots.txt file is a standard text file placed in the root directory of a website that tells search engine robots and automated crawlers which URLs they can or cannot access on your site.

Does robots.txt protect affiliate funnels from ad review crawlers?

No. While legitimate search engine bots obey robots.txt, ad review crawlers (Meta, TikTok, Google Ads) and malicious scrapers often ignore disallow directives. Active server-side edge filtering with Cloaker.net is required to block unauthorized crawlers.

Does a disallow rule in robots.txt prevent a page from being indexed?

Not necessarily. Disallowing a URL stops search bots from crawling its content, but if external links point to that URL, Google can still index the URL itself. To ensure a page is never indexed, use a noindex meta tag instead.

Where should the robots.txt file be uploaded?

The robots.txt file must be uploaded to the root directory of your website domain (e.g., https://example.com/robots.txt).

Works seamlessly with your favorite ad networks, trackers, and traffic sources