Site Owner FAQ

How do I verify it's really Minifetch.com?

Real Minifetch.com requests come from the following IP address: 138.68.38.1 AND we identify ourselves with this user-agent: minifetch/1.0 (+https://minifetch.com/site-owner-faq)
User-agents can be spoofed, so the IP address is your best verification.

Does Minifetch respect my robots.txt?

Yes. By default, we fetch as our minifetch user agent from a fixed IP, we honor your robots.txt, following both wildcard (*) rules and any rules that name minifetch. If a page is disallowed for minifetch, we don't fetch it. The request returns an error instead of data, and we don't charge for it.

However a fetch is routed, we observe your crawl-delay and cache the result, so Minifetch won't put repeated or rapid load on your server.

We offer a free "Check URL" button (and API endpoint) for users to see if a URL is accessible or blocked to minifetch by robots.txt before spending credits to fetch it. Website owners can also use this feature to check if minifetch is allowed to fetch pages on their site(s). We cache that response so our users aren't abusing your robots.txt file, either.

If you don't want us fetching something on your site, you can either:

- Block all robots (including us):
User-agent: *
Disallow: /my-private-page
- Block only minifetch:
User-agent: minifetch
Disallow: /my-private-page
- To explicitly allow minifetch while blocking other bots:
User-agent: minifetch
Allow: /

User-agent: *
Disallow: /
- Or to allow us with a crawl delay:
User-agent: minifetch
Crawl-delay: 1
Allow: /
Why allow Minifetch?

• Checks your robots.txt: our minifetch bot reads it before fetching
• Easy on your server: we observe your crawl-delay and cache results, on every fetch
• Transparent identity: a named user agent paired with a published IP you can verify
• On the record: response data says how the fetch was made and whether robots.txt was obeyed
• Cost barrier: micropayments deter abuse and spam

What does Minifetch actually do?

Minifetch is a hosted toolkit of web page extraction primitives. Run them as a full technical SEO audit or call one at a time for a fraction of the price — and a fraction of the tokens. No subscription.

Explore the API →

Install the Minifetch API client →

How is Minifetch different from other services?

• By default, we fetch as an identified bot — a named user agent paired with a published IP site owners can verify — and obey robots.txt, including crawl-delay.
• Every response carries a proxy block recording how the fetch was made: the user agent we sent and whether robots.txt was obeyed, so how data was gathered travels with it.
• We cache results and honor rate-limits to keep load off the sites we fetch.
• Pay-per-request pricing keeps usage deliberate and deters spam and abuse.
• We respond to site owner & user feedback.

How we fetch web pages

Minifetch extracts publicly available metadata and content from pages reachable without authentication or JavaScript execution. By default it fetches as an identified user agent/ bot from a fixed IP and obeys robots.txt, including crawl-delay.

Every response carries a proxy block: the minifetch user agent we sent and whether robots.txt was obeyed on the fetch. Proof of how the fetch happened, not just a promise, for regulated use-cases where provenance matters.

Future add-ons:

• Residential proxies for hard-to-reach pages
• JavaScript execution
• Access to authenticated or logged-in content

What about "noindex/nofollow" robots meta tags on individual pages?

We surface those in our audits and data, but we don't obey them. Here's why:

• Robots meta tags in html pages tell search engines what to index
• We're not a search engine - we're an on-demand fetch service
• We work like a browser visiting a page, not a crawler indexing your site
• By default, Minifetch respects robots.txt (crawling rules), but page-level noindex/nofollow meta tags control indexing, which we don't do

Is Minifetch open-source?

Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the NPM community for 10+ years. During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.

Can I contact Minifetch?

If you have questions or need assistance:
join our Discord server.

To leave feedback or join the bulk credits waitlist:
use our feedback form.

While you're at it, give us a follow on X.com (Twitter):
@minifetch