Minifetch is a hosted toolkit of web page extraction primitives. Run them as a full technical SEO audit or call one at a time for a fraction of the price — and a fraction of the tokens. No subscription.
In the browser, we offer a free "Check URL" button (and API endpoint) to see if a URL is likely
accessible or blocked to our minifetchbot by robots.txt before spending credits to fetch it.
We cannot guarantee a URL will be accessible just based on robots.txt. A site may block access for many reasons. Checking robots.txt first is a reasonable step before fetching to ensure success.
If you own the website in question, we have a tutorial to help you unblock Minifetch: How To Allow Minifetch to Fetch Your Pages
Websites can publish a robots.txt file listing pages that are off-limits to bots.
By default, we fetch as an identified minifetch bot from a fixed IP and obey those rules, so a
disallowed URL comes back as a "blocked by robots.txt" message (a 502 with no data).
You're never charged for a blocked page.
Response data also carries a proxy block showing how the fetch was made
and whether robots.txt was obeyed. That gives you provenance you can keep, which matters
if you work in a regulated space or need an auditable record of how your data was gathered.
Want to check before you spend credits? Our free "Check URL" button (and API endpoint) tells you whether a URL is likely accessible or blocked first. For full policies, see our Terms of Service.
In a browser, the "Go Fetch!" button on the homepage fetches and runs an SEO audit on the target URL you enter into the search box.
If you are signed in to your account, a successful "Go Fetch!" will be deducted from your credit balance. You can sign up for an account and get started for free by visiting the dashboard.
If you do not wish to sign up for an account, you will be prompted to use your browser plugin wallet to pay in USDC cryptocurrency on Coinbase's Base network or Solana. This service uses Coinbase's x402 protocol to facilitate these payments.
Either way you pay as you go. There is no Minifetch account setup or monthly fee. We do not charge for blocked pages or pages that error.
Yes, full API documentation is available →
We also provide API docs in LLMs.txt format for your AI agents.
For coders, the Minifetch API client on npm has a good README for getting setup quickly in your Javascript/ Typescript project.
We surface those in our audits and data, but we don't obey them. Here's why:
proxy block recording how the fetch was made: the user agent we sent and whether robots.txt was obeyed, so how data was gathered travels with it.Minifetch extracts publicly available metadata and content from pages reachable without authentication or JavaScript execution. By default it fetches as an identified user agent/ bot from a fixed IP and obeys robots.txt, including crawl-delay.
Every response carries a proxy block: the minifetch user agent we sent and whether robots.txt was obeyed on the fetch. Proof of how the fetch happened, not just a promise, for regulated use-cases where provenance matters.
Future add-ons:
Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the NPM community for 10+ years. During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.
minifetch-apiNEW — the official Minifetch API client for Javascript/ Typescript
npm install minifetch-api --save
url-metadata — the open-source npm package that powers Minifetch
npm install url-metadata --save
Both from the same creator with 4 Million+ downloads over 10 years.
Add them to your project with confidence.
If you have questions or need assistance:
join our Discord server.
To leave feedback or join the bulk credits waitlist:
use our feedback form.
While you're at it, give us a follow on X.com (Twitter):
@minifetch