How to Allow Minifetch to Fetch Your Pages

This tutorial is for site owners. Minifetch checks every website's robots.txt before fetching its pages and respects whatever rules are set. If your site blocks all bots by default — which is common — users won't be able to fetch your pages through Minifetch. This guide shows you how to selectively allow Minifetch while keeping other bots blocked, and how to verify it's working.

Building with an AI agent? This tutorial is also available as a skill your AI agent can load directly: minifetch.com/skills/unblock-minifetch/SKILL.md

How Minifetch identifies itself

Minifetch sends the following user-agent string with every request:
```
minifetch/1.0 (+https://minifetch.com/site-owner-faq)
```
Minifetch matches on the minifetch token, so any User-agent directive containing minifetch (case-insensitive) will be picked up correctly.

If your robots.txt is missing or returns an error, Minifetch defaults to allowed. If it returns a status code 403, 418, or 429, Minifetch treats the entire site as blocked.
Allow Minifetch while blocking all other bots

Add the following to your robots.txt. Order of blocks doesn't matter — Minifetch's parser matches on the most specific user-agent rule:
```
User-agent: minifetch
Allow: /

User-agent: *
Disallow: /
```
This explicitly grants Minifetch access to all pages while blocking every other crawler.
Allow Minifetch on specific paths only

To restrict Minifetch to certain sections of your site, use path-level rules:
```
User-agent: minifetch
Allow: /blog/
Allow: /products/
Disallow: /

User-agent: *
Disallow: /
```
Set a crawl delay

Minifetch strictly observes the Crawl-delay directive (value in seconds). Use it to slow Minifetch down — or to speed it up below the default so audits of your own site finish faster. For example, to let Minifetch fetch your pages twice as fast as the default:
```
User-agent: minifetch
Allow: /
Crawl-delay: 0.5
```
Here Crawl-delay: 0.5 tells Minifetch it may fetch a page every half-second instead of the default one second — halving the time to crawl a batch of your URLs. Fractional (sub-second) values are honored.

Without any Crawl-delay set, Minifetch defaults to 1 second between requests to your domain, so fetching 10 URLs takes at least 10 seconds. This default is deliberate — Minifetch never hammers your server or slows it down for your real users — and you can adjust it up or down as you like. Minifetch caches robots.txt for 24 hours, so changes may take up to a day to propagate.
Block Minifetch entirely

To block Minifetch along with all other bots:
```
User-agent: *
Disallow: /
```
Or to block Minifetch specifically while keeping other bots allowed:
```
User-agent: minifetch
Disallow: /

User-agent: *
Allow: /
```
Verify your robots.txt is working

After updating your robots.txt, use the free preflight endpoint to confirm Minifetch can fetch your pages correctly:

From the homepage, enter your URL in the search box and click "Check URL".

Or with the minifetch api client:
```
await client.preflightCheck("https://example.com/your-page");
```
Or from your cli:
```
curl "https://minifetch.com/api/v1/free/preflight/url-check?url=https://example.com/your-page"
```
A successful allow response looks like this:
```
{
  "success": true,
  "results": [
    {
      "data": {
        "url": "https://example.com/your-page",
        "allowed": true,
        "crawlDelay": 1
      }
    }
  ]
}
```
If allowed is still false after updating, confirm your robots.txt is accessible at https://example.com/robots.txt and has been re-deployed. Minifetch caches robots.txt for 24 hours, so changes may take up to a day to propagate.

How Minifetch identifies itself

Allow Minifetch while blocking all other bots

Allow Minifetch on specific paths only

Set a crawl delay

Block Minifetch entirely

Verify your robots.txt is working