# Minifetch Skill: Allow Minifetch Access via robots.txt

This skill is for **site owners** who want to explicitly allow Minifetch to crawl
their pages while keeping other bots blocked.

**Base URL:** https://minifetch.com

---

## How Minifetch Reads robots.txt

Minifetch checks your robots.txt before every fetch using your site's user agent rules.
It identifies itself with the following user agent string:

```
minifetch/1.0 (+https://minifetch.com/site-owner-faq)
```

Minifetch matches on the `minifetch` token, so any `User-agent` directive containing
`minifetch` (case-insensitive) will be picked up correctly.

If your robots.txt is missing or returns an error, Minifetch defaults to **allowed**.
If it returns a 403, 418, or 429, Minifetch treats the entire site as **blocked**.

---

## Allowing Minifetch While Blocking All Other Bots

Add the following to your `robots.txt`. The order of blocks does not matter --
Minifetch's parser matches on the most specific user-agent rule:

```
User-agent: minifetch
Allow: /

User-agent: *
Disallow: /
```

This explicitly grants Minifetch access to all pages while blocking every other crawler.

---

## Allowing Minifetch on Specific Paths Only

To restrict Minifetch to certain sections of your site:

```
User-agent: minifetch
Allow: /blog/
Allow: /products/
Disallow: /

User-agent: *
Disallow: /
```

---

## Blocking Minifetch Entirely

To block Minifetch along with all other bots:

```
User-agent: *
Disallow: /
```

Or to block Minifetch specifically while allowing other bots:

```
User-agent: minifetch
Disallow: /

User-agent: *
Allow: /
```

---

## Setting a Crawl Delay

If you want to allow Minifetch but limit how frequently it crawls your site,
add a `Crawl-delay` directive (value in seconds):

```
User-agent: minifetch
Allow: /
Crawl-delay: 10
```

Minifetch strictly observes crawl delays. Without one set, it defaults to 1 second between requests.

---

## Verifying Your robots.txt

After updating your robots.txt, you can verify Minifetch can see it correctly
using the free preflight endpoint:

```
curl "https://minifetch.com/api/v1/free/preflight/url-check?url=https://yoursite.com/your-page"
```

A successful response will show:
```json
{
  "success": true,
  "results": [
    {
      "data": {
        "url": "https://yoursite.com/your-page",
        "allowed": true,
        "crawlDelay": 1
      }
    }
  ]
}
```

If `allowed` is still `false` after updating, check that your robots.txt is
accessible at `https://yoursite.com/robots.txt` and has been re-deployed.
Minifetch caches robots.txt for 24 hours, so changes may take up to a day to propagate.

---

## Questions?

Visit our [Site Owner FAQ](https://minifetch.com/site-owner-faq) for more detail
on how Minifetch identifies itself, what it does and does not crawl, and our
ethical scraping practices.


---

## Contact
Questions or need help? Join our [Discord server](https://discord.gg/EM6ET8Dshm).

Feedback or bulk credits waitlist? Use our [feedback form](https://forms.gle/rkMi7T23bHJc8XFw9).

Follow us on X: [@minifetch](https://x.com/minifetch)

Full API docs: https://minifetch.com/llms.txt
All skills: https://minifetch.com/SKILL.md