User FAQ

Trying out Minifetch?

Here's what you need to know.

What does Minifetch actually do?

Search, scrape, extract & SEO audit web pages. Minifetch is a hosted toolkit for developers and AI agents. Always pay-per-fetch, no subscription.

Explore the API →

Install the Minifetch API client →

What does the "Check URL" button do?

Minifetch offers two ways to fetch a page: the native Minifetch proxy or a rotating external proxy. The free "Check URL" button (and API endpoint) helps you choose before spending credits: it checks whether a URL is allowed or blocked for our minifetch bot by robots.txt.

Allowed: use the cheaper native Minifetch proxy.
Blocked: if the site is not yours, use the external proxy routes. If it is yours, see How To Allow Minifetch to Fetch Your Pages.

We cannot guarantee a URL will be accessible just based on robots.txt. A site may block access for other reasons. The choice is always yours, per fetch: Minifetch never switches proxies on your behalf.

Does Minifetch respect robots.txt?

That is your choice, per fetch. Minifetch offers two ways to fetch a page, and the API route you call decides which one runs:

Native Minifetch proxy: fetches as an identified minifetch bot from a fixed IP and respects robots.txt, including crawl-delay. A disallowed URL comes back as a "blocked by robots.txt" message (a 502 with no data).
External proxy: an external rotating proxy that does not check robots.txt. It reaches pages the native one can't.

Response data carries a proxy block showing which proxy made the fetch and whether robots.txt was obeyed. That gives you provenance you can keep, which matters if you work in a regulated space or need an auditable record of how your data was gathered. For those use-cases, stay on the native Minifetch proxy.

You're never charged for a blocked page on either. Want to check before you spend credits? Our free "Check URL" button (and API endpoint) tells you whether the native Minifetch proxy is allowed to fetch a URL. For full policies, see our Terms of Service.

How do I pay for a "Go Fetch!"?

In a browser, the "Go Fetch!" button on the homepage fetches and runs an SEO audit on the target URL you enter into the search box.

If you are signed in to your account, a successful "Go Fetch!" will be deducted from your credit balance. You can sign up for an account and get started for free by visiting the dashboard.

If you do not wish to sign up for an account, you will be prompted to use your browser plugin wallet to pay in USDC cryptocurrency on Coinbase's Base network or Solana. This service uses Coinbase's x402 protocol to facilitate these payments.

Either way you pay as you go. There is no Minifetch account setup or monthly fee. We do not charge for blocked pages or pages that error.

Are there API Docs?

Yes, full API documentation is available →

We also provide API docs in LLMs.txt format for your AI agents.

For coders, the Minifetch API client on npm has a good README for getting setup quickly in your Javascript/ Typescript project.

What about "noindex/nofollow" robots meta tags on individual pages?

We surface those in our audits and data, but we don't obey them. Here's why:

Robots meta tags in html pages tell search engines what to index
We're not a search engine - we're an on-demand fetch service
We work like a browser visiting a page, not a crawler indexing your site
By default, we respect robots.txt (crawling rules), but page-level noindex/nofollow meta tags control indexing, which we don't do

How is Minifetch different from other similar services?

By default, we fetch as an identified bot — a named user agent paired with a published IP site owners can verify — and obey robots.txt, including crawl-delay.
Every response carries a proxy block recording how the fetch was made: the user agent we sent and whether robots.txt was obeyed, so how data was gathered travels with it.
On the native Minifetch proxy, we cache results and honor crawl-delay to keep load off the sites we fetch.
Pay-per-request pricing keeps usage deliberate and deters spam and abuse.
We respond to site owner & user feedback.

How we fetch web pages

Minifetch offers two ways to fetch a page: the native Minifetch proxy or a rotating external proxy. The native Minifetch proxy is identified, respects robots.txt and is cheapest. The external proxy reaches pages the native one can't. You choose per fetch, by choosing the API route for your use-case. Built to be orchestrated by agents and humans who want maximal control.

	Native Minifetch proxy	External proxy
Routes	`/extract`	`/proxy/extract`, `/proxy/render/extract`
Fetches as	Identified `minifetch` user agent from a fixed IP	External rotating proxy (webscrapingapi.com), optional `country`
robots.txt	Respected, including crawl-delay	Not checked
JavaScript	Not executed	Executed on `/proxy/render`
Cached	Yes	No, every call is a fresh fetch
Price	Cheapest	Native price plus a surcharge, see Proxy Routes

Decide before you pay. The free /preflight/url-check tells you whether the native Minifetch proxy is allowed to fetch a URL.
Proof per request. Every response carries a proxy block: which proxy fetched the page, the user agent sent and whether robots.txt was obeyed. For regulated use-cases where provenance matters, stay on the native Minifetch proxy.
No charge for blocked or errored pages on either.

Not currently available on either proxy: authenticated or logged-in content.

Is Minifetch open-source?

Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the NPM community for 10+ years. During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.

minifetch-apiNEW — the official Minifetch API client for Javascript/ Typescript

npm install minifetch-api --save

url-metadata — the open-source npm package that powers Minifetch

npm install url-metadata --save

Both from the same creator with 4 Million+ downloads over 10 years.
Add them to your project with confidence.

Can I contact Minifetch?

To leave feedback or join the bulk credits waitlist:
use our feedback form.

If you have questions or need assistance:
join our Discord server.

While you're at it, give us a follow on X.com (Twitter):
@minifetch