Site Owner FAQ

Noticed Minifetch.com in your logs?

Here's what you need to know.

How do I verify it's really Minifetch.com?

Real Minifetch.com requests come from the following IP address: 138.68.38.1 AND we identify ourselves with this user-agent: minifetch/1.0 (+https://minifetch.com/site-owner-faq)
User-agents can be spoofed, so the IP address is your best verification.

Does Minifetch respect my robots.txt?

Our minifetch bot does. The native Minifetch proxy fetches as the minifetch user agent from a fixed IP and honors your robots.txt, following both wildcard (*) rules and any rules that name minifetch. If a page is disallowed for minifetch, we don't fetch it. The request returns an error instead of data, and we don't charge for it. On these fetches we observe your crawl-delay and cache the result, so Minifetch won't put repeated or rapid load on your server.

Minifetch also offers users a second way to fetch: an external rotating proxy (webscrapingapi.com). Those fetches do not come from our fixed IP or user agent, do not check robots.txt and are not cached. Users choose per fetch, and every response records which proxy was used and whether robots.txt was obeyed.

We offer a free "Check URL" button (and API endpoint) for users to see if a URL is accessible or blocked to minifetch by robots.txt before spending credits to fetch it. Website owners can also use this feature to check if minifetch is allowed to fetch pages on their site(s). We cache that response so our users aren't abusing your robots.txt file, either.

If you don't want us fetching something on your site, you can either:

- Block all robots (including us):
User-agent: *
Disallow: /my-private-page
- Block only minifetch:
User-agent: minifetch
Disallow: /my-private-page
- To explicitly allow minifetch while blocking other bots:
User-agent: minifetch
Allow: /

User-agent: *
Disallow: /
- Or to allow us with a crawl delay:
User-agent: minifetch
Crawl-delay: 1
Allow: /
Why allow Minifetch?

Checks your robots.txt: our minifetch bot reads it before fetching
Easy on your server: we observe your crawl-delay and cache results, on every native fetch
Transparent identity: a named user agent paired with a published IP you can verify
On the record: response data says how the fetch was made and whether robots.txt was obeyed
Cost barrier: micropayments deter abuse and spam

What does Minifetch actually do?

Search, scrape, extract & SEO audit web pages. Minifetch is a hosted toolkit for developers and AI agents. Always pay-per-fetch, no subscription.

Explore the API →

Install the Minifetch API client →

How is Minifetch different from other services?

By default, we fetch as an identified bot — a named user agent paired with a published IP site owners can verify — and obey robots.txt, including crawl-delay.
Every response carries a proxy block recording how the fetch was made: the user agent we sent and whether robots.txt was obeyed, so how data was gathered travels with it.
On the native Minifetch proxy, we cache results and honor crawl-delay to keep load off the sites we fetch.
Pay-per-request pricing keeps usage deliberate and deters spam and abuse.
We respond to site owner & user feedback.

How we fetch web pages

Minifetch offers two ways to fetch a page: the native Minifetch proxy or a rotating external proxy. The native Minifetch proxy is identified, respects robots.txt and is cheapest. The external proxy reaches pages the native one can't. You choose per fetch, by choosing the API route for your use-case. Built to be orchestrated by agents and humans who want maximal control.

	Native Minifetch proxy	External proxy
Routes	`/extract`	`/proxy/extract`, `/proxy/render/extract`
Fetches as	Identified `minifetch` user agent from a fixed IP	External rotating proxy (webscrapingapi.com), optional `country`
robots.txt	Respected, including crawl-delay	Not checked
JavaScript	Not executed	Executed on `/proxy/render`
Cached	Yes	No, every call is a fresh fetch
Price	Cheapest	Native price plus a surcharge, see Proxy Routes

Decide before you pay. The free /preflight/url-check tells you whether the native Minifetch proxy is allowed to fetch a URL.
Proof per request. Every response carries a proxy block: which proxy fetched the page, the user agent sent and whether robots.txt was obeyed. For regulated use-cases where provenance matters, stay on the native Minifetch proxy.
No charge for blocked or errored pages on either.

Not currently available on either proxy: authenticated or logged-in content.

What about "noindex/nofollow" robots meta tags on individual pages?

We surface those in our audits and data, but we don't obey them. Here's why:

Robots meta tags in html pages tell search engines what to index
We're not a search engine - we're an on-demand fetch service
We work like a browser visiting a page, not a crawler indexing your site
By default, Minifetch respects robots.txt (crawling rules), but page-level noindex/nofollow meta tags control indexing, which we don't do

Is Minifetch open-source?

Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the NPM community for 10+ years. During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.

minifetch-apiNEW — the official Minifetch API client for Javascript/ Typescript

npm install minifetch-api --save

url-metadata — the open-source npm package that powers Minifetch

npm install url-metadata --save

Both from the same creator with 4 Million+ downloads over 10 years.
Add them to your project with confidence.

Can I contact Minifetch?

To leave feedback or join the bulk credits waitlist:
use our feedback form.

If you have questions or need assistance:
join our Discord server.

While you're at it, give us a follow on X.com (Twitter):
@minifetch