How to Allow Minifetch to Fetch Your Pages

This tutorial is for site owners. Minifetch checks every website's robots.txt before fetching its pages and respects whatever rules are set. If your site blocks all bots by default — which is common — users won't be able to fetch your pages through Minifetch. This guide shows you how to selectively allow Minifetch while keeping other bots blocked, and how to verify it's working.

Building with an AI agent? This tutorial is also available as a skill your AI agent can load directly: minifetch.com/skills/unblock-minifetch/SKILL.md

How Minifetch identifies itself

Minifetch sends the following user-agent string with every request:
```
minifetch/1.0 (+https://minifetch.com/site-owner-faq)
```
Minifetch matches on the minifetch token, so any User-agent directive containing minifetch (case-insensitive) will be picked up correctly.

If your robots.txt is missing or returns an error, Minifetch defaults to allowed. If it returns a status code 403, 418, or 429, Minifetch treats the entire site as blocked.
Allow Minifetch while blocking all other bots

Add the following to your robots.txt. Order of blocks doesn't matter — Minifetch's parser matches on the most specific user-agent rule:
```
User-agent: minifetch
Allow: /

User-agent: *
Disallow: /
```
This explicitly grants Minifetch access to all pages while blocking every other crawler.
Allow Minifetch on specific paths only

To restrict Minifetch to certain sections of your site, use path-level rules:
```
User-agent: minifetch
Allow: /blog/
Allow: /products/
Disallow: /

User-agent: *
Disallow: /
```
Set a crawl delay

Minifetch strictly observes the Crawl-delay directive (value in seconds). Use it to slow Minifetch down, or to speed it up so audits of your own site finish faster. For example, to let Minifetch fetch your pages twice as fast as the default:
```
User-agent: minifetch
Allow: /
Crawl-delay: 0.5
```
Here Crawl-delay: 0.5 tells Minifetch it may fetch a page every half-second instead of the default one second, halving the time to crawl a batch of your URLs. Fractional (sub-second) values are honored.

Without any Crawl-delay set, Minifetch defaults to 1 second between requests to your domain, so fetching 10 URLs takes at least 10 seconds. This default is deliberate so Minifetch never hammers your server or slows it down for your real users, and you can adjust it up or down as you like. Minifetch caches robots.txt for 24 hours, but you don't have to wait for an edit to take effect: the verify step below shows how to force an immediate re-check with &fresh=true.
Block Minifetch entirely

If you wish to block Minifetch along with all other bots:
```
User-agent: *
Disallow: /
```
Or to block Minifetch specifically while keeping other bots allowed:
```
User-agent: minifetch
Disallow: /

User-agent: *
Allow: /
```
Verify your robots.txt is working

After updating your robots.txt, use the free preflight endpoint to confirm Minifetch can fetch your pages correctly.

From the your browser, enter the following URL in the address bar. Be sure to use the &fresh=true parameter to refresh the cache:
```
https://minifetch.com/api/v1/free/preflight/url-check?url=https://example.com/your-page&fresh=true
```
Or from your CLI. Add &fresh=true to bypass Minifetch's 24-hour robots.txt cache and check against your just-edited file right away:
```
curl "https://minifetch.com/api/v1/free/preflight/url-check?url=https://example.com/your-page&fresh=true"
```
Or with the minifetch api client:
```
await client.preflightCheck("https://example.com/your-page", { "fresh": true });
```
A successful allow response looks like this:
```
{
  "success": true,
  "results": [
    {
      "data": {
        "url": "https://example.com/your-page",
        "allowed": true,
        "crawlDelay": 0.5,
        "minifetchCache": {
          "hit": false,
          "cachedAt": "2026-08-16T18:22:10.000Z",
          "expiresAt": "2026-08-17T18:22:10.000Z"
        }
      }
    }
  ]
}
```
The minifetchCache object confirms which robots.txt Minifetch used: "hit": false means it just re-fetched your live file (what you want right after an edit), while "hit": true means it served a cached result.

If allowed is still false after updating, confirm your robots.txt is accessible at https://example.com/robots.txt and has been re-deployed. Without &fresh=true, Minifetch caches robots.txt for 24 hours, so an unforced check can take up to a day to reflect your change. Use &fresh=true to confirm it right away.

How Minifetch identifies itself

Allow Minifetch while blocking all other bots

Allow Minifetch on specific paths only

Set a crawl delay

Block Minifetch entirely

Verify your robots.txt is working